Oracle Database Appliance Patch Update error when default gateway is not pingable - DCS-10001 and PRGO-1664

When setting up an Oracle Database Appliance (ODA) the configuration of the network with odacli configure-firstnet does check, if the default gateway is pingable. If it is not, the configuration is not possible. 

But one of our customers (or better the network administrators) later configured the environment (router), that pings are not responded, which means, they time out (this is part of the security strategy of this company). The network does work, but the ODA can not ping the default gateway anymore (no one can).

As we tried to patch an ODA from ODA Release 19.21 to 19.24 we therefore run into an ugly error while the Grid Infrastructure was upgraded from 19.21 to 19.24, leaving us with two unzipped home directories:

"DCS-10001:Internal error encountered: Failed to set ping target on public network."

If you take a look into the dcs-agent.log, you can find a more detailled error message:

DEBUG [Server patching : JobId=<something>] [] c.o.d.a.r.s.g.GiOperations: ping failed: unknown gateway <IP-Address of Gateway>

The creation of the prepatch report for the server was fine ([root@oda1 opt]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.24.0.0.0). It finished successful without error, so we didn't expect that we run into an error later on. 

But while running the server patch ([root@oda1 opt]# /opt/oracle/dcs/bin/odacli update-server -v 19.24.0.0.0) we got the above error. 

We asked the network team to fix the ping problem, tested the ping manually and restarted the odacli update-server -v 19.24.0.0.0 a second time. Sometimes, when an ODA update stucks, it is fine to just restart the update-server again, but this time we got another error:

 “DCS-10001:Internal error encountered: Failed to patch GI with RHP : DCS-10001:Internal error encountered: PRGO-1664 : The specified source working copy \"OraGrid192100\" is a software-only working copy...”

Thanks god we did an "odabr backup -snap" before we started the update and by running "odabr restore -snap -force" we were able to restore the "before" image of the operating system stuff. If you start odabr restore without the force option, you will get a warning "Clusterware patchlevel at backup time is not what you have now. ODABR can't be considered a tool to perform Grid Infrastructure downgrade" and the snap will not be restored.

Unfortunately, the grid infrastructure is not part of the odabr backup with newer ODA software releases, which means, we still had two Grid Infrastructure homes and the problem with the working copy as a software only working copy. 

Searching in support.oracle.com does not really help, searching for PRGO-1664 does not result in useful hits. But there is a small section in the note for the odabr tool (2466177.1). "if a GI update was in progress and you have done a restore", it may happen that the executable permissions are set wrong. 

To fix this, set the GRID_HOME to the old grid environment as user root and run [root@oda1]# GRID_HOME/crs/install/rootcrs.pl -lock

This will change the executable permissions - in our case in the 19.21 grid directory back and after this change patching including the odacli update-server -v 19.24.0.0.0 was running fine. 

Depending on the time you did the odabr backup, you need to start with patching the dcs stack or running the prepatch-report creation again. 

For the next ODA software versions, there is an enhancement request placed for the development, so the gateway ping check is then part of the prepatch report and will not lead to a corrupted GI update.

In the meantime it could be helpful to test manually, if the default gateway is pingable. To find out the gateways ip address you can run a "ip route show" command. It outputs a line "default via <IP Address>". This ip address is your gateway - a "ping <IP Address>" will (hopefully) work. If so, you will not run into this problem, we got. We will add the ping test to our manual check list, it is a 10 second work and it will save an hour work later on.


 

An underestimated feature: Using Oracle Remote Listener with Standard Edition 2 Licenses - Part 1 (Setup)

Even for small and medium-sized companies that work with Standard Edition 2 (SE2) databases from Oracle, high available and secure systems are a “must have”. How to install a remote listener is covered in this post (more details on high availability and security are covered in the next posts in the near future).

The databases are typically protected with additional software for high availability. The best-known provider is certainly dbvisit with its product “dbvisit Standby”, but there are other products available like the one from the company I work for - this alternative standby software is called robotron*Standby.

All software providing high availability for SE2  work more or less in the same way: a database is permanently recovered from the archived redo logs of the primary in a second environment and thus kept up to date. The applications or users connect (now with pluggable) databases via a TNSNames.ora entry, JDBC connect or easy connects. The host and the listener (ports) for the connections then typically change, if a switchover or failover happens.
The application database service (I really recommend DON'T use the default PDB service for your applications) remains the same in terms of name and is only available where the primary database is running and read/write opened, but there are still a few things to note.

Typically, connection entries for such highly available databases then look like this in TNSNames.ora (analogous in JDBC or with easyconnect):

application_service_name_ha=
 (DESCRIPTION=
   (ADDRESS_LIST=
     (LOAD_BALANCE=off)
     (FAILOVER=on)
        (ADDRESS=(host = proddbhost)(protocol = tcp)(port = 1521))
        (ADDRESS=(host = standbydbhost)(protocol = tcp)(port = 1521))
   )
  (CONNECT_DATA=
   (SERVICE_NAME=application_service_name.domain)
  )
 )

As the default values are not sufficient, additional parameters are often specified for the failover, e.g.TRANSPORT_CONNECT_TIMEOUT, CONNECT_TIMEOUT, RETRY_DELAY or RETRY_COUNT. I don't want to describe all of them here in detail, but in case you need them you have at least heard of them. 


However, the problems that result from this type of connecting to the application service of the database are always the same: you have to store both database hosts in the connect string and the connection failover to the other machine then always takes a moment.
Anyone using JDBC or easyconnect syntax will have a hard time with these complicated connection strings anyway. There are also some applications in which such connection strings cannot be captured due to the length of the string (especially if someone combines more than ONE Standby with a primary database). And last, but not least, the local listener at the database is "open for the whole world", right? So anyone who knows the database servers host name and the standard port can try to access the database.

Customers who use Enterprise Edition licenses with Data Guard are provided with a tool called Connection Manager (CMAN), this tool can be used as a proxy function/proxy listener, which does not only handles to forward connections, but also allows to secure connections/servers/services a little bit more. However, as the Connection Manager is only part of the Enterprise Edition database license, Standard Edition 2 customers must find another solution.

Remote listeners are the best possible solution to create some kind of connection manager by your own. These remote listeners are often installed on systems that have to run “anyway” for an application to work. Typically, these are the application servers, printer servers, etc., which are known to the end user and without an application is not or only partially available. In principle, however, any server that is itself highly available is suitable.

To be able to use a remote listener, a Full Oracle Database Client is required on this server, an Instant Client does NOT work for this. The client can be downloaded e.g. via https://edelivery.oracle.com.
As most listeners are compatible downwards for older releases, the highest version can theoretically be used here. Nevertheless, most of my customers want to use the same release that is also used for the database to stay in sync.

Be aware, that downloading from https://edelivery.oracle.com means one gets the BASE Release (e.g. 19.3.0.0) of the client. It can (and should) be patched (not covered in this blog post). 

After the download, the client must be installed on the server as an administrator - installing on Windows means setting  registry entries and new Windows services are created. Here are some hints:
It is important (for security reasons) that only used components are selected for installation. Setting “Custom” as installation type let one choose the right parts. Most administratos will probably select “Use Windows Built-in Account” for the Oracle Home/Windows Services user, which is fine. The software location is either based on the Oracle standard or it does make sense to use an application server software directory (e.g. create an oracle subdirectory at your application software base directory. It is sufficient to select the “Oracle Net Listener” - dependent components are installed automatically.
However, SQLDeveloper or ODBC driver or other things often required on the application server for installing or updating the application, so this can also be selected here.

Installer Selection Screen

After the installation is done, one can find new services in Windows - the only one that must run is the listener service. All other services installed by the client can be disabled or set to manual startup type.

The listener is started automatically, but there is no service registered, because the configuration is missing. 

Listener status without configuration

The Listener.ora file must be adapted so that a database can now register with the Listener.
As always, this file is located under ORACLE_HOME\network\admin. The listener line with EXTPROC1521 as listening endpoint is usually removed  for security reasons, it is not needed.
Now two parameters are added, whereby the “LISTENER” in the parameters stands for the name of the listener (which can of course also have a different name):


VALID_NODE_CHECKING_REGISTRATION_LISTENER = ON
REGISTRATION_INVITED_NODES_LISTENER = (<IP-Addresses>,<HostNames>,<IP-WITH-WILDCARDS>)

Typically, a 2-node environment for high availability would also include both nodes. The listener.ora on the application server appserver1.localdomain could look like this:

# listener.ora Network Configuration File: C:\app\oracle\product\19.0.0\client_1\NETWORK\ADMIN\listener.ora
# Generated by Oracle configuration tools.

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = appserver1.localdomain)(PORT = 1521))
    )
  )

VALID_NODE_CHECKING_REGISTRATION_LISTENER = ON
REGISTRATION_INVITED_NODES_LISTENER = (server1.localdomain,server2.localdomain)

After the configuration is saved, the Listener configuration needs to be reloaded (lsnrctl reload listener or e.g. restart the service).

Now everything is prepared at the remote listener configuration to allow service registration from databases hosted on server1.localdomain and server2.localdomain, the database(s). The next step is to set the remote listener parameter at the CDB to reach the remote listener. In this example one can run on the database the following command:

alter system set remote_listener=' (ADDRESS = (PROTOCOL=TCP)(HOST=appserver1.localdomain)(PORT=1521))' scope=both;

Now the database registers itself with the remote listener.

 
In addition, the tnsnames.ora (or a jdbc connection string) can be apapted to connect to the database using the application server host listener instead of the database host listener:

jsotestpdb,jsotestpdb.localdomain =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = appserver1.localdomain)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = jsotestpdb.localdomain)
    )
  )

The (application) client does not know anything about the database host at the time of connection.

Which impact does this have for standby databases? What about more "security" after setting up a remote listener successful? What happens with PDB Services while cloning to other databases?

Valid questions that will be answered in the next posts. Stay tuned.

How to start preparing your database for the upgrade to Oracle database 23ai - even without an on-premises released database software!

 As many of you already know, Oracle's autoupgrade tool will be the only supported option for database upgrades in the future. This fact can also be found in the support notes of the 23ai database.

However, the autoupgrade tool - and most people don't know this - already supports the upgrade to Oracle database version 23ai in Release 24 (available since roundabout March of this year). The question therefore arises, how can you TODAY use the autoupgrade tool to optimally prepare your existing databases for the upgrade to 23ai ? Especially now, when the perhaps quiet time of the year is approaching, but the official release date of 23ai has been postponed to “sometime in calendar year 2025” (postponed once again)?

The preparation

First of all, the latest version of the autoupgrade.jar must be downloaded from Oracle Support and can be found at Document ID 2485457.1.
This is currently a build from October 2024 (24.7.241021). The autoupgrade.jar can next be copied to the server and tested for functionality. If no Java is installed on the server, the JDK from the highest Oracle Home release can be used. This has always worked so far.
Oracle's support for direct upgrades to 23ai is limited to 19c and 21c databases, this means older database versions cannot be prepared for a direct upgrade to 23ai using the autoupgrade tool. An intermediate upgrade (to 19c) is therefore necessary beforehand or, for example, data migration with Datapump Export-Import is a possible way to move from older database versions to 23ai. However, no pre-test can be one with datapump.

If the execution of autoupgrade.jar (java -jar autoupgrade.jar -version) works on the server, the next step is to create a config file for the database that is to be prepared for the 23ai upgrade.

The following information is important here:

  • the log directories (global and per DB)
  • Which SID should be tested/analyzed
  • The source Oracle Home - this needs to be a 19c or 21c database home
  • The target Oracle Home - is set, but this directory does not exist - the database is not yet available on-premises
  • Start_time and upgrade_node can be set by default (NOW/localhost)
  • Important - the autoupgrade tool normally takes the target_version from the target home. However, as this does not exist, it is important to set 23 fixed as target_version here.
autoupgrade config file  
Config file



 Analyzing the existing database

Once this preparation has been completed, the autoupgrade tool can now be run in Analyze mode against the source database. NO changes are made to the source database, this means this procedure is completely safe.
However, it is important that all pluggable databases (PDB) are open. PDBs that are in mount status are not analyzed.

Warning - PDB is mounted, not open

If all PDBs are open, there is no longer a warning and the corresponding autoupgrade job starts analyzing the existing database.

Job starts analyzing the database without warning

The “status” command can be used to display the progress of the analysis of all jobs on the Autoupgrade CLI. It shows 4 databases (1 CDB + Seed PDB + 2 user PDBs).

Status of all autoupgrade jobs running

To see the details of the individual databases, just add the job with number to the “status” command. Now one can see the stage progress per container (the 4 databases).


The job should be successfully completed after a few minutes. There are typically no errors in the analysis phase.

Job completed

The analysis result (in various formats, including JSON, text and HTML) is now available for the analyzed database in the Prechecks subdirectory of the log directories for the job. The next picture does show the HTML.

Overview HTML

The first overview already shows that there are components in the 19c database that would be removed during an upgrade to 23ai. These are remnants of Oracle Multimedia - marked yellow.

If one now looks in the detailed areas of the databases (CDB and PDBs), one will find valuable information on how to deal with such changed or removed components. The Autoupgrade tool recommends taking a look at various support notes, for example.

Notes to look at support.oracle.com documents

In the database analyzed here (which was already upgraded from a 12c environment), the autoupgrade tool also found a configuration of traditional auditing. Traditional auditing was first supplemented by Oracle some time ago with Unified Auditing and then replaced - or is now officially desupported for the first time, so that Unified Auditing must be switched to.
As the configuration of auditing can be a very time-consuming process, it is advisable to adapt the configuration on the existing database even before the upgrade planning officially starts after 23ai.

Finding Traditional Auditing

The autoupgrade tool may already find things as part of the POST check, things which then need to be  done on the 23ai database, such as changing the LOB type at an audit table. Of course, this cannot yet be prepared in the existing environment, but it does provide an indication of any manual work that may need to be done (and maybe any additional downtime) during a later upgrade.

Post Check - LOB Change on Audit Table

Incidentally, even though the screenshots are now all from a Linux environment, the analysis method was also tested with a 19.8 database under MS Windows. It should therefore be possible to use this procedure on all supported operating systems.

Now start prepare your database for a successful upgrade to database version 23ai.



Oracle database client installation "The java.library.path system variable is missing or invalid"

As I heard it more than once in the last months, it is the right time to write a blog post about it. I still don't know, why it happens so often at the moment (except of a lack of reading the documentation), but I know, how it will (hopefully) work for you, if you run in this problem (without reading the documentation).

The typical environment where this happen is a Red Hat Linux (mostly 8.x and 9.x), but it can happen on any Linux distribution.

The customers of mine all tried to install a 19c 64-bit Client on a Red Hat Linux system as the installer stopped somewhere with this message:

Preparing to launch Oracle Universal Installer from /tmp/OraInstall20XX-XX-XX_XX-XX-XXPM. Please wait ... The java.library.path system variable is missing or invalid. Please set java.library.path with a correct value and retry the operation.

Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class oracle.sysman.oii.oiip.oiipg.OiipgPropertyLoader

First to check is always, if the java environment is set up/available correctly, but typically, it is. Second is to check the documentation for the operating systems requirements of the product you want to install - in case of 19c clients on Linux, the documentation can be found here.

In the above case, the error message  "The java.library.path system variable is missing or invalid" is just wrong. Java is fine and available and the java.library.path is correct.

In this case and with the documentation you can then check the supported (kernel) version and afterwards the Linux system for the required packages, this is e.g. the list for Red Hat Linux 8.x:
bc, binutils, elfutils-libelf, elfutils-libelf-devel, fontconfig-devel, glibc, glibc-devel, ksh, libaio, libaio-devel, libXrender, libX11, libXau, libXi, libXtst, libgcc, libnsl, librdmacm, libstdc++, libstdc++-devel, libxcb, libibverbs, make, policycoreutils, policycoreutils-python-utils, smartmontools, sysstat.

There is also a section in the documentation which allows to check the optional packages required for some products, e.g. ipmiutil (for Intelligent Platform Management Interface), libnsl2 (for Oracle Database Client only), libnsl2-devel (for Oracle Database Client only), net-tools (for Oracle RAC and Oracle Clusterware), nfs-utils (for Oracle ACFS).

Now, what happend to all the customers with the above error message while installing the client? As we found out step by step, one of the required packages were not installed as X86 64bit version: The libnsl. But this isn't told by the installer - it complains about the java.library.path system variable.

All customers just installed libnsl.x86_64 as newest version using 

yum install libnsl.x86_64

and then they retried the installation of the client succesfully. All other packages, except libnsl, were setup with the Red Hat installation, only libnsl was missing.

If you run an Oracle Enterprise Linux you can use the preinstall-rpm for the database (oracle-database-preinstall-19c) - by the way even if you only want to install the client - to get all the libs installed automatically. If you are on Red Hat, you really should check the libraries according to the documentation.

Yes, it is a (mostly) stupid work to compare libs, but if it prevents from a whole day searching for an error with a misleading error message...