Autoupgrade.jar on MS Windows: Database is open but "Error Cause: The database <--> is currently open with status of CLOSED."

People are using autoupgrade.jar to upgrade the Oracle databases more and more, this is the good news.

But the more people using autoupgrade.jar - the more things arise, while working with it. That's the bad news (or maybe it's also good news, because we do get a better feeling if things happen, WHY they happen).

At the blog of my employer, one was asking for a solution of the problem that autoupgrade in analyze mode is reporting a status of closed, but the database is open. He is connecting with an application and sqlplus, tnsping, etc. is working.

He was running autoupgrade on a 12c database on a MS Windows server in analyze mode, when he hits this error, also with the latest downloaded version of autoupgrade.jar (you should always download the latest autoupgrade.jar from Oracle Support).

The error was similar to this one:

Unsupported language [de], defaulting to English
AutoUpgrade tool launched with default options
------------ ERROR ------------
Error Cause: The Database orcljso2 appears to be down or open with the incorrect binaries for the mode ANALYZE. Ensure it is open with C:\app\oracle\product\12.2.0\dbhome_1

------------ ERROR ------------
Error Cause: The database orcljso2 is currently open with a status of CLOSED.  For ANALYZE mode, it needs to be open in one of the following: [OPEN, MOUNTED].

Unable to connect to database orcljso2 for entry upg1

To be honost, this is NOT an autoupgrade error, because there are some constraints for autoupgrade you have to fullfil, so it can do its work. One of the main things is, autoupgrade does connect to the database using "sqlplus / as sysdba". There isn't any tnsnames.ora or password involved. If this doesn't work in your environment, you need to fix that. It is also not related to MS Windows, but in Linux environments, typically connecting "/ as sysdba" is working (as you do everything using your "oracle" user and the default groups).

But how to know, what is the root problem, as autoupgrade does only tells you "the database is closed"?

Autoupgrade does write very, very detailed logs (or better traces). As you have specified a log directory for autoupgrade, you can find the logs there.

There are 2 possibilities where autoupgrade does put the logs into. If you don't specify a local entry, it puts it into the global directory, otherwise in the local one of your database specification in the config file.

• global.autoupg_log_dir=C:\app\oracle\autoupgrade
• upg1.log_dir=C:\app\oracle\autoupgrade\orcljso2
  

In my environment, I can then find the log files "autoupgrade.log" and "autoupgrade_err.log" in the directory C:\app\oracle\autoupgrade\cfgtoollogs\upgrade\auto

Some examples, of what can go wrong while (not) connecting with "/ as sysdba" and what "autoupgrade.log" does show as output.

1. A missing part in the database home of the source

I have specified

• upg1.source_home=C:\app\oracle\product\12.2.0\

while the right entry would be

• upg1.source_home=C:\app\oracle\product\12.2.0\dbhome_1

The entry at the log file then looks like

2020-07-09 14:52:45.844 ERROR Invalid value for source_home [C:\app\oracle\product\12.2.0\dbhome1] - Utilities.validateDirectory
2020-07-09 14:53:19.251 ERROR Unexpected IOException [null] [SELECT STATUS FROM SYS.V$INSTANCE;] - ExecuteSql$SQLClient.run
java.io.IOException: Cannot run program "C:\app\oracle\product\12.2.0\bin\sqlplus" (in directory "C:\Users\JOERG~1.SOB\AppData\Local\Temp"): CreateProcess error=2, The System cannot find the File specified
    at java.lang.ProcessBuilder.start(Unknown Source)
    at oracle.upgrade.commons.processes.ExecuteProcess.startSqlPlusProcess(ExecuteProcess.java:315)
    at oracle.upgrade.commons.sql.ExecuteSql$SQLClient.run(ExecuteSql.java:882)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: CreateProcess error=2, The System cannot find the File specified
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    ... 4 more

2020-07-09 14:53:19.259 ERROR The database orcljso2 appears to be down or open with the incorrect binaries for the mode ANALYZE,
ensure it is open with C:\app\oracle\product\12.2.0 - UpgradeConfigDBValidator.initializeIsDBUp
2020-07-09 14:53:19.262 ERROR The database orcljso2 is currently open with a status of CLOSED.  For ANALYZE mode, it needs to be open in one of the following: [OPEN, MOUNTED]. - UpgradeConfigDBValidator.initializeIsDBUp

Solution: Correct the Source home path in your configuration. I think the same error can happen, if the Source Home is not the same as the home the service was started from.

Note: If you specify e.g. the home and miss the "_", you do get another error message:

Invalid value for source_home [C:\app\oracle\product\12.2.0\dbhome1]
It was not possible to validate the directory for source_home of entry upg1)

2. Wrong SID in the configuration file

If you have specified a wrong SID in the configuration file, the error is a little bit hidden.

2020-07-16 14:33:53.786 ERROR The database orcljso appears to be down or open with the incorrect binaries for the mode ANALYZE,
ensure it is open with C:\app\oracle\product\12.2.0\dbhome_1 - UpgradeConfigDBValidator.initializeIsDBUp
2020-07-16 14:33:53.798 ERROR The database orcljso is currently open with a status of CLOSED.  For ANALYZE mode, it needs to be open in one of the following: [OPEN, MOUNTED]. - UpgradeConfigDBValidator.initializeIsDBUp
2020-07-16 14:33:53.803 INFO Content of the file C:\app\oracle\product\12.2.0\dbhome_1\sqlplus\admin\glogin.sql is:
--
-- Copyright (c) 1988, 2005, Oracle.  All Rights Reserved.
--
-- NAME
--   glogin.sql
--
-- DESCRIPTION
--   SQL*Plus global login "site profile" file
--
--   Add any SQL*Plus commands here that are to be executed when a
--   user starts SQL*Plus, or uses the SQL*Plus CONNECT command.
--
-- USAGE
--   This script is automatically run
-- - GLoginLogger.logGlogin

So it halts at this state in the log. The only thing where you can see that you have specified a wrong SID is at the error itself (my database is ORCLJSO2 and not ORCLJSO).

Solution: Correct the SID in your configuration file.

Note: If the SID is correct but the dbname is wrong, the autoupgrade tool connects to the database and starts working.

Wrong configuration is:

    upg1.dbname=orcljso
    upg1.sid=orcljso2

Output of the autoupgrade.log at job level is:

   DataBase Name:orcljso2
   Sid Name     :orcljso2

3. Wrong SQLNET configuration or ORA_DBA group missing

Another reason, why the autoupgrade tool cannot connect to the database is that you have some values at your sqlnet configuration, which don't allow this or your (windows shell) user is not part of the ORA_DBA group.

As example, you have specified:

SQLNET.AUTHENTICATION_SERVICES = (NONE)

The output at the logfile is pretty same as for a wrong SID:

2020-07-16 15:06:19.805 ERROR The database orcljso2 appears to be down or open with the incorrect binaries for the mode ANALYZE,
ensure it is open with C:\app\oracle\product\12.2.0\dbhome_1 - UpgradeConfigDBValidator.initializeIsDBUp
2020-07-16 15:06:19.821 ERROR The database orcljso2 is currently open with a status of CLOSED.  For ANALYZE mode, it needs to be open in one of the following: [OPEN, MOUNTED]. - UpgradeConfigDBValidator.initializeIsDBUp
2020-07-16 15:06:19.825 INFO Content of the file C:\app\oracle\product\12.2.0\dbhome_1\sqlplus\admin\glogin.sql is:
--
-- Copyright (c) 1988, 2005, Oracle.  All Rights Reserved.
--
-- NAME
--   glogin.sql
--
-- DESCRIPTION
--   SQL*Plus global login "site profile" file
--
--   Add any SQL*Plus commands here that are to be executed when a
--   user starts SQL*Plus, or uses the SQL*Plus CONNECT command.
--
-- USAGE
--   This script is automatically run
-- - GLoginLogger.logGlogin

Solution: Check your SQLNET configuration, e.g. set SQLNET.AUTHENTICATION_SERVICES = (NTS), start "lusrmgr" to check the the user account is part of the ORA_DBA group. Be aware, that there can be also some kind of "ORA_<ORACLE_HOME>_DBA" in the "lusrmgr"!

As it is a little bit more special using MS Windows, the documentation for authentication can be found here.

If you use a domain account, have a special look at the documentation part "Overview of Operating System Authentication Enabled at Installation". You need to put this domain account into ORA_DBA AND grant local administrative rights.

Hope that helps you running autoupgrade.jar (even on MS Windows) successful.

Register of a standby database fails (within Oracle GI/repository) on Oracle Database appliance

A couple of days ago I tried to register a standby database on ODA with a newer relase of ODA (in my case 18.8) and Grid Infrastructure. I followed the procedure of the Dbvisit Standby documentation.

In there, it is described that you need to start the database on the standby environment as "read only".

Afterwards, you should be able to register the database in the Grid Infrastructure using odacli.

The procedure is described at this manual in detail and should work with Dbvisit Standby Version 8 and 9.

Unfortunately, if you run the ODACLI to register the database, it fails.

odacli register-database -c OLTP -s odb6 -t SI -sn <mydb.mydbdomain.com> -p <somepassword>

The job at the ODA is started, but it errors out:

Job details
----------------------------------------------------------------
ID: 432c4c54-0d2f-XXXX-XXXX-aa15f27ea3ab
Description: Database service registration with db service name: <mydb.mydbdomain.com>
Status: Failure
Created: May 5, 2020 3:40:06 PM CEST
Message: DCS-10001:Internal error encountered: Failed to archive log all
Session altered.

alter system archive log current
*
ERROR at line 1:
ORA-01109: database not open

This also means, that you see the "failed" configurations in odacli list-databases.

I have created a Service Request with Dbvisit to check the manual/the process of registering the database on ODA. At the meantime, the only solution is to open the database "read/write" to register it at the ODA.

To open the standby read/write it's necessary to "switchover" the primary to the standby you want to register and run odacli register-database on the -now- primary. Afterwards you can switch the roles of the databases again, so your original standby is a standby again at the end.

For your application that means, that it is necessary to have a small maintenance window where the end users can't work.

I will edit this blog post as soon as I do have more information, so come back regularly if you are interested in any news.

Autoupgrade.jar - Skip Fixups Step-by-Step

Autoupgrade will be more and more important to upgrade Oracle databases to the newer releases (18c, 19c, 20c). Mike Dietrich e.g. has told at the DOAG conference 2019, that autoupgrade will be the future of Oracle Exadata database upgrades also.

The autoupgrade tool is a very quick evolving tool and it becomes more stable and more helpful with every new release (nearly every quarter, sometimes every 3 or 4 weeks, you can download a new version from support.oracle.com).

But there are still things, you should know and which can be misunderstood at the documentation, if you start first with the autoupgrade tool. And then you may don't get the result(s) you are expecting.

One of the things I write today about is how you can skip automatically running fixups.This is e.g. useful if you want to prepare the database with the pre-upgrade part and run the autoupgrade.jar later on at a specific time (e.g. at the weekend or late at the evening) and not in dialog mode.This means, you spent less time at the "upgrade" phase itself (e.g. part of the normal prefixups is gathering statistics).
It is also helpful if you run autoupgrade.jar with "deploy" or "upgrade" mode, because BOTH modes automatically run POST fixups (and you may want to skip one or more and do them (later) manually).

Like everytime, first step is to download the latest autoupgrade.jar from MyOracleSupport!

Then, you check the java environment and create your own configuration file. (A small how to (see preparation at https://a-different-view-by-js.blogspot.com/2019/12/everything-goes-auto-1.html) can be found within my slide deck from DOAG 2019 "Alles wird Auto": https://a-different-view-by-js.blogspot.com/p/blog-page.html or at Mikes Blog https://mikedietrichde.com/2019/06/13/create-and-adjust-the-config-file-for-autoupgrade-19c/).

I typically use "create_sample_file config" and change that to fit my needs.

After everything is specified in the config file, the autoupgrade.jar is started in analyze mode.

 After some minutes the job has finished. Now it is essential to note down the job number, because you will need it later on in your configuration file. 

 

At the specified log path, you have a subfolder <job-number>/prechecks with a couple of files.

First is, check carefully ALL the issues at the preupgrade-HTML report, because you want to find the ones you don't want the autoupgrade.jar let fix "automatically" (in pre or post phase) while running later on. 

Then, you need to edit the fixup file in the same directory. It's the <database>_checklist.cfg file. 

E.g. if you want to skip the "DICTIONARY_STATS" fixup, you have to change the part at [runfix] from YES (default, as you can see in the picture) to NO and save your file.

Now have a look at the documentation for overriding the default fixups. The documentation clearly states "By default, AutoUpgrade uses the most recent generated file." So this means, if you run the autoupgrade tool later on with the "deploy" mode, the new job creates a new config file and does not recognize yours automatically!

If you use the autoupgrade mode "upgrade" then the prefixup phase is completely skipped, so also the fixups you want to get done will not run.

You have to create a new entry inside of your autoupgrade-configuration.
The parameter is called "checklist" and there you can specify the checklist configuration with the modified prefixup steps.
I like to copy/paste that from another default configuration file I do have, so I only need to uncomment the last line and put the right directory/filename into the configuration.

So the configuration for checklist in my file after the change is:

upg1.checklist=c:\oracle\autoupgrade\logs\ECKES\ECKES\100\prechecks\eckes_checklist.cfg

I typically fix also some other things manually, so I let run the analyze phase more than once and skip all the manually fixed things in the eckes_checklist.cfg of my FIRST job. This makes it easier to handle and I don't need to remember which is the fixup skip configuration file I have specified in my autoupgrade configuration. 

Last step is to run the autoupgrade again in "deploy" mode. It will take the checklist configuration file specified in your config instead of a new one, makes the fixups in there, skips the one with "NO" and proceeds with the upgrade, makes the post fixups which are not set to "NO" and finishes the upgrade.

Isn't that easy in comparison with all the manual things you had to do before when upgrading your database?

Everything goes Auto? First experiences with the Oracle Autoupgrade-Tool - Part 5 - DBA still matters...

As this post is a part of a series of posts regarding Autoupgrade-tool, you might want to start with the first (configuration) or the second post (pre-upgrade analysis).

As every other DBA, I am not error free. Any good administrator is doing things wrong sometimes, this is normal, especially if you do a lot stuff in a very short time and you are very busy.

Well, to be honest, things happened also because I made some mistakes. E.g. at one of my tests (with GRP) I didn't used an administrator command for running the autoupgrade tool in MS Windows. MS Windows was quitting that with an OpenSCManager failed, Access denied error. (Slide 48).
As I tried to use "restore" in the console, Windows again throwed ne the error. What should I do now? I am lost in the middle of nowhere (ok, I could have went back to my VM snapshot, but I wanted to test the tool).
I exited the console (it was warning me that 1 job will stop) (Slide 49) and started the autoupgrade tool again in a administrator command window.

It attached automatically to the old job and I started a restore again. I don't know if there was something still in an undefined state, but first nothing happens.
Starting the "restore" a second time was successful. I have watched the job status saying it's flashing back parallel with the alert.log and after a while it was back on the pre-upgrade state (Slide 50-54).
That's a very robust behaviour, even if the first restore seemed to do nothing (I think it was able to fix a broken state, so the second execution could succeed).

A second error I did was arising while using the "add_after_upgrade_pfile" parameter (Slide 55). I had a stupid spelling mistake in that file. So autoupgrade finished, but the database was not started.
A look at the detailled log file and I have seen that the pfile wasn't copied / changed to spfile due to this error. I then copied the corresponding init.ora to the dbs/database subdirectory, created a spfile out of it and everything was fine. No need to roll back using the GRP...

If you are on MS Windows, one last thing happened while creating the Windows service in the new Oracle Home with oradim. I was lazy and just copied the whole command from the preupgrade.html.
The DIM-00003 error I got doing this is misleading.
There is no missing parameter - you can't specify the spfiles name after the "-spfile" switch (Slide 57).

My personal summary (Slide 58):

Always, always, always download the newest version of the autoupgrade-tool. As it evolves and improves fast, check again often, if there is a new version of it.

Well, "everything automatic" is something relative. You can still run in some hassles (as I did on MS Windows), you maybe try things which are non-production-use and so on - not only at the autoupgrade-tool things can happen, but also at the database level or because you as administrator are doing something wrong (like the most of my examples show). By the way, there was a whole session at the DOAG conference what can happen (from a database perspective) during an upgrade (and how you can find/fix the errors).

The tool is hands free, yes, but personally I would prefer to keep an eye on it while using it in production the first times. As always: If you test your configuration well in your environment and everything works there, then it works also in production! This is, where I used the VM snapshots in my test environments a lot to test as much as I can in a short timeframe.

What I was really impressed about is, how stable and robust a tool is, that is not that long on the market, how good resume and restore are working and, again, how detailled the logs are.

As we do have a lot customers at Switzerland running Windows with (Swiss) German or (Swiss) French language, we need to test the newer versions of the autoupgrade tool. 
Linux environments of our customers are all in english and also don't have the MS Windows service problems, so it's easier for us to use the tool there. If you have more than 1 or 2 databases and these are similiar (especially the environment), it does really make sense to step into using the tool.

Hope you enjoyed the series. I will make some more tests next year. By the way, Mike Dietrich told at his session, that autoupgrade.jar will be the tool for upgrading exadata databases in the future!

Everything goes Auto? First experiences with the Oracle Autoupgrade-Tool - Part 4 - Non-CDB to PDB conversion after upgrade

As this post is a part of a series of posts regarding Autoupgrade-tool, you might want to start with the first (configuration) or the second post (pre-upgrade analysis).

To convert a non-CDB to a PDB automatically after the upgrade, you can set "target_cdb" (first create an empty CDB or use an already existing one) in your configuration file.

While testing that, I was getting an error that the guaranteed restore point couldn't be dropped. But it wasn't existing, as I have set "restoration=no" in my configuration (Slide 42, 43). Well, I opened another SR that this step fails on Windows (on Linux I only had already CDBs, therefore I couldn't test that there). Mike Dietrich than told me that this feature is shown on his blog, but it does not exists in the documentation. As it is not official released yet, it should show the future of the tool.

As Mike is right, this is not a scenario for a SR,  I closed mine immediately (Slide 44), for sure, there is no issue with a non-production-use of the tool. Only 2 or 3 days (!) later (now speaking with the development directly) I got a newer pre-version from Mike personally! So I was able to go forward with my tests.

The autoupgrade.jar tool then stepped over the GRP problem but was running into a plug-in violation (Slides 45/46). The, again, very good logfiles recorded an error, that the description.xml could not be read. The description.xml file did exist, so was it another non-production-use thing?
Development helped me analysing it and so we found out together that the problem seems to be related to the database version 19c (on Windows). If we put the description.xml into ORACLE_HOME/database the file can be found, but not at it's original place (Slide 47).
While the one thing is database bound, the fixes for the GRP will for sure be seen in the December or January release of the tool.


Edit (December, 2nd): What happened after the DOAG conference: With development feedback and some long testing runs over the weekend I have figured out, where the problem starts (it's a Windows/Oracle Database rights issue). I am waiting for the developers assessment on that, maybe I write another, different post, which is only a little bit related to autoupgrade.jar as the problem may hit you at every MS Windows environment.

Edit (May 14th, 2020): The newest version of autoupgrade.jar does now support Non-CDB to PDB conversion officially. I have not tested that yet.

The next post for my DOAG session can be found here.

Everything goes Auto? First experiences with the Oracle Autoupgrade-Tool - Part 3 - Deploy mode

As this post is a part of a series of posts regarding Autoupgrade-tool, you might want to start with the first (configuration) or the second post (pre-upgrade analysis).

Most of the tests I did in the deploy mode, because I used snapshots (instead of backups) a lot while testing, so it was (also with SE2) easy to revert to the last state. Snapshots have been my best friends as I didn't want to wait for the restore/recovery...

As I said, I was sometimes a little bit confused that with "lsj" e.g. status showed "finished" while message said "starting" (Slide 26), so I preferred the detailed
status -job <jobno> to look at the status of the job.
There are a lot details you can see, e.g. the operation, what is already done, how many stages are pending, etc., the status command at a database with CDB architecture also shows all PDBs (including seed) which are in touch by the tool (Slide 27 and Slide 28).

As the first tests without archivelog mode succeeded, I tried to use "restoration=YES" in my configuration. This means, you use a guaranteed restore point (GRP, Enterprise Edition only) where you can roll back the database to, if something goes wrong.

For using GRP you need a Fast Recovery Area (FRA). Most of our customers don't use FRA by default, so this is something we need to introduce in our case, exclusively and only for the time of the upgrade.

Testing with GRP, after some minutes, I have seen with another "status" command, that there is a warning. The recovery area is nearly full and the upgrade could "appear hung" if the recovery area is filled up (Slide 29).
To find out, what happens, if the FRA is full, I started later on another autoupgrade run with a far too small FRA (Slide 30).
Thankfully, the tool did not hung, but throwed an unexpected exception error. In the very detailed logs (it's more a debug trace than a normal logfile) you can easily find the error: "MIN_RECOVERY_AREA_SIZE" is to small.

Fixing that can be done online with "alter system", but only with scope=MEMORY. The database is restarted a couple of times with a pfile while it runs through the deploy mode. (Slide 31)
You can either wait for this error happen more than once or, what I did, change the different pfiles, e.g. "during_upgrade_pfile_catctl.ora" (which is located at the dbupgrade subdirectory of your job) in addition. After resuming the job (Slide 33), the autoupgrade finished fine (Slides 34, 35, 38).

As the size of the FRA can be critical, it's good to set it big enough before you begin with the deploy mode. You could, e.g. use the "before_action" parameter in your configuration to change the FRA size or you can use add_during_upgrade_pfile, for example. (Slide 32)

While the tool is running, you can see with the status command that it's e.g. upgrading CDB$ROOT, upgrading PDB$SEED or your user PDBs (in parallel) (Slides 34/35).
There is also a "tasks" command (Slide 36) where you can see all the tasks that are running or waiting, but I didn't see anything there I think I would need at my daily work.

If you run the autoupgrade.jar on MS Windows, a temporary service in the target Oracle Home is created (Slide 37), dropped afterwards and you have to drop the old original service and create the new one manually, that's fine.
When the upgrade has finished, don't forget to drop the GRP by yourself, otherwise the FRA will be filled up completely (you could set a parameter in your configuration to do this automatically, but I prefer to test the upgraded environment first) (Slide 38).

On MS Windows I had an unexpected error again on my german environment as stopping the service for the database to be upgraded was successful, but the tool was raising an error (Slide 40). I have opened another SR for that and I think it will be fixed soon (there is a bug open for that). After installing the US language pack and changing the Oracle user account to "US" language, everything worked fine.

The next post will be about the Non-CDB to PDB conversion with the autoupgrade tool.

Everything goes Auto? First experiences with the Oracle Autoupgrade-Tool - Part 2 - Pre-Upgrade-Analysis

This is the second post to the first experiences with the Autoupgrade tool. Please start with the first one.

You start the pre-grade analysis with the parameters -config <yourconfigfile> -mode analyze in console mode (Slide 20).

With "lsj" you can see the job number and some information, like stage, operation, status and message. As I was not able to understand all the different status combinations at the output, I started to prefer to select only the job number and issue a
"status -job <jobno>" to get the detailed information regarding a job.
The analysis job finishes after some minutes (1 to 3 depending on my virtualbox on the laptop or the VMWare server) (Slide 21).

At your autoupgrade log directory, you then get a number of files, the two I want to mention here are the preupgrade.html file and the checklist.cfg file (Slide 22).

As not all servers do have a web browser installed (mostly on linux there isn't a GUI), you really should copy the html file to your desktop!

There is soooo many information in this html and you should treat all messages serious, independend if they are "errors" or only "warnings", "recommended" or "infos" (Slides 23-25).

What you have to look at is what is the topic that raises you a message and if there is a fixup available. Fixup available yes means, the autoupgrade tool can correct this topic before (or after) the database is/was upgraded automatically by itself.

For our "customer-like" environment we had e.g. things in the dba recyclebin. This can be fixed automatically, but as DBA I like to prefer to have a look into the recycle bin first.Maybe there is something in there we already need?

Another topic found was 10G password version users are existing - they can't connect to the 19c database later on! Will be really bad if you don't fix that by yourself.
The "streams configuration" found in the database wasn't used, so I dropped that manually (the html shows you the DBMS-Packages you need as output).

Again, check all the things you see at the html carefully! Then decide to let it be fixed automatically (if there is a fixup available), to change the configuration or to ignore the issue (like ignoring some invalid procedures in our application schema depending on the type of software in use).

If you want to ignore a fix, you can tweak the checklist.cfg file by changing runfix from YES to NO. Keep in mind to NOT run the autoupgrade tool afterwards with the "deploy" parameter. You need to run the fixup and the upgrade modes manually.
Deploy creates a new pre-analysis, fixes things and upgrades the database in another job and does not know of things you may have changed in the older checklist.cfg which belongs to an old, finished job.

Another topic, we often see: Some of us do have underscore parameters at the spfile and the tool detects this and says, there is no fixup available. This is true, but there is a workaround.
If you want to get rid of all underscore parameters you can set a global parameter (remove_underscore_parameters=yes) and you can add the once you really need later with the add_to_pfile-files you can specify. Otherwise you can ignore this and proceed with the fixups or with the deploy mode (the underscore parameter are still in your spfile after the upgrade).

Everything goes Auto? First experiences with the Oracle Autoupgrade-Tool - Part 1 - introduction and configuration

At the DOAG conference 2019 I had a presentation (due to the filled up room it was repeated the day after) regarding the Oracle Autoupgrade tool. The slides are in german and can be found at my Slide Download Center. As most slides are screenshots only, I will write in this series of posts a little bit about this new tool.

The first part will now cover an introduction and how to setup the configuration file.

Introduction:
After OOW 2018 I have heard the first time of the Autoupgrade Tool and I was very curious. I have downloaded the slides and what I have found was a thing I thought it should be there since years.
A tool which helps you upgrading your databases from one release to another without running all these scripts, pre-installation.jars and DBUA manually. With less hands on and with a lot of things automated. The more I have read the more curious I got.

Later on, I have seen autoupgrade as part of the documentation of Oracle 19c, but I wasn't really able to see how it could work and I also had no time to step into it in detail.

Then, Mid of June 2019, I have seen Mike Dietrichs (Mister Upgrade Guru) blog post about how to create a sample configuration file and with how less parameters an upgrade could be started.
So I decided (after a short conversation with Mike on twitter, June 13th) to test the tool and to send a proposal for a presentation at the DOAG conference. I already had installed a 19c Database software on my laptop with Windows 10, so I decided to make the first small test there.

Unfortunately, I've got a java exception at running autoupgrade.jar with a -version flag. Also the downloaded version from support.oracle.com had this problem. (Slide 11) It's so typical, that it hits me...😳

So I was really sorry to inform Mike that I couldn't test that due to (in my opinion) a problem with the german environment I had. I opened a SR to pass all the logs to the Oracle Support and Mikes Team.
July 1st (! I started testing mid of June! - Slide 13) I've got an email from Mike that the bug should be fixed in the version they have uploaded some days before. The fix included a "fallback" to default language (english) if the OS language is not supported. I downloaded that version and yes, it was working now. (Slide 15)

I asked my colleague to set up a "Swiss customer similar database environment" on a Windows 2012 server and I prepared a virtualbox with Oracle Linux and some CDBs.
As I was busy over the summer I only were able to test things in my spare time mostly. So I've started with reading the documentation and Mikes blog.
The more I read, the more I have seen some targets, that my employer could reach by using this tool (Slide 7):

As an ISV and System Integrator, we have a lot of databases to upgrade (most of them from 12.1 to 19c) next year. Some on Linux, some on Windows, some on ODA (but ODA is out of scope here).
The database editions are Standard Edition 2 and Enterprise Edition and we have all kind of licenses (Embedded, Application Specific and Full Use).
Main target for us is to get rid of any manual work as this is most error-prone, e.g. changing SPFile parameters, switching from Non-CDB to the new CDB architecture, maybe combining application upgrades with database upgrades (scripted), etc..

After the virtualbox and the Windows environments (on VMWare) have been installed, I was ready to start with the tests.

Preparation:

As a lot servers don't have (the right) java (version) installed, it's best to use the java from the ORACLE_HOME/jdk/bin directory of 18c or 19c (Slide 15). This works fine. Also, as release cycles of the autoupgrade tool are very short, you should download always the newest autoupgrade.jar from support.oracle.com. Even if you have checked the note a week ago, check again, before you start your work.

With that in mind, you can run the tool with the -create_sample_file config parameter (by the way, on my german environment it says unsupported language, defaulting to English) (Slide 16).

It's very easy to use the sample config file (Slide 17) to create your own configuration. A small thing that can be found on Windows is the global log directory misses "backslashes". This does not work out of the box but is easy to fix.On Linux, the default log directory is fine and points to /home/oracle. (Slide 18)

Next to do is to specify (Slide 19) e.g. the DB_UNIQUE_ID as "DBNAME", the "SID", "SOURCE HOME" and "TARGET HOME", "TARGET_VERSION" of the database and, if you wish so, even more (see the documentation for all parameters). The "START_TIME" parameter can be set to 'now' or a date/time - the format can be found in the documentation (no NLS_LANG setting). With "add_after_upgrade_pfile" (or delete or before or during instead of after) you can change automatically the SPFILE content.

One of the things I really like is to change is the tag "UPG1." to a tag that gives more information (e.g. EDM for our Energy Data Management DB, DAPHNE for the museums DB, etc.) (Slide 19).

So you can create one config file for all kind of databases and they are pretty named. Running "UTLRP" for recompilation is a nice thing, also have a look at the parameter for "TIMEZONE_UPG"rades as it is often forgotten in manual upgrades. You should also know the "RESTORATION" parameter, more information on this parameter will follow later.

If you have setup the config file, you are ready to start to work with your database(s).


The next step is the Pre-Upgrade analysis (next post).

Next public events where you can meet me

Hi,

more events to come - here you can find me speaking in the next months (unfortunately, all presentations will be held in german language):

•  The first presentation is at the 15th Robotron Business Cafe at Dübendorf where I will speak about operating applications in clouds.
• In May (14th) you can find me at the Oracle Fokustag Datenbank of Robotrons HQ at Dresden, there I will present Oracle Database 18c/19c New Features (including XE) and what a DBA need to prepare to run 19c as a long-term release.
• The next event is the big Swiss Oracle User Group Day which takes part at May, 22nd at Olten. I will speak there again about 19c New Features. You can find me at Track 5 in the morning (10.30h) at the Fachhochschule Nordwestschweiz at Olten. 
• The last event in May (28th at Stade de Suisse) is some kind of after work beer. Not a meetup, but an event regarding PostgreSQL and Oracle Database Appliance (ODA). Yes, you can consolidate also open source databases together with e.g. Standard Edition 2 or Enterprise Edition databases on the Oracle Database Appliances (most companies do have free resources at the ODAs they own). We will tell you how you can run PostgreSQL on ODA, which nice features you can use and what you might consider running open source databases on ODA.

So there are enough events where you can meet me - no excuse if you miss ALL of them 😉

And by the way - HAPPY EASTER TO EVERYONE!

Some more presentations... part 2 (with some corrections)

Hi all,

I have some date changes to tell regarding the post about where I do speak next about Oracle stuff.
So the new (and I hope fixed) dates in the next months are:

• 26.03.2019, Oracle Fokustag at Robotron Schweiz. I will speak about 19c New Features and a little bit about the new version of DBSAT (Database Security Assesment Tool), I think.
  
  Agenda und Anmeldung: Fokustag Oracle CH-Wil (St. Gallen)

• 9./10.04.2019, Oracle Database Appliance Events at Stade de Suisse, Bern, and at Oracles Smart Innovation Center at the Prime Tower Zurich. Main focus there is how the ODA can help you on premise to find your way to the cloud and why Standard Edition 2 fits perfect to Oracles Database Appliance.
  
  Agenda und Anmeldung: Oracle Database Appliance Workshop Bern
  
  Agenda und Anmeldung: Oracle Database Appliance Workshop Zuerich

• New: 07.05.2019 Robotron BusinessCafe at Duebendorf. I will speak a little bit about Oracle Cloud and (maybe) in addition a little bit about DBSAT (Database Security).
   
• CHANGED: Then, in May (14th), I will speak at the Oracle Fokustag of Robotrons HQ at Dresden, also about 18c/19c New Features (including XE) and what a DBA need to prepare to run 19c as a long-term release.
  
  Agenda und Anmeldung (bald möglich): Fokustag Datenbanken in Dresden

• And last, but not least, the big Swiss Oracle User Group Day takes part at May, 22nd at Olten. I will speak there again about 19c New Features.
  
  

Some more presentations...

Hi all,

so many new things, so many testing, so many work to do at the moment.
But despite the fact I am testing Oracle Autonomous TP Cloud and PostgreSQL (last week I have finished the tests with pgBackRest as Robotron Standard Backup tool for our PostgreSQL customers) I am planning for some more presentations at events at Switzerland and Germany.

At the moment the following events are confirmed, unfortunately, for the english speaking people, they are all in german language.

• 26.03.2019, Oracle Fokustag at Robotron Schweiz. I will speak about 19c New Features and a little bit about the new version of DBSAT (Database Security Assesment Tool), I think. 

• 9./10.04.2019, Oracle Database Appliance Events at Stade de Suisse, Bern, and at Oracles Smart Innovation Center at the Prime Tower Zurich. Main focus there is how the ODA can help you on premise to find your way to the cloud and why Standard Edition 2 fits perfect to Oracles Database Appliance. 

• Then, in May (7th), I will speak at the Oracle Fokustag of Robotrons HQ at Dresden, also about 18c/19c New Features and what a DBA need to prepare to run 19c as a long-term release.

• And last, but not least, the big Swiss Oracle User Group Day takes part at May, 22nd at Olten. I will speak there again about 19c New Features. 

Hope to see you at one or more events!

Single Tenant ist unnütze Komplexität? / Single Tenant is only unnecessary complexity? - Relocate a PDB on-the-fly

Two parts of my three part blog posts are already available, so this is the third and (at the moment) last post about Single Tenancy and what you can do easily with pluggable databaseses using database links.

What you do need to use the scripts as they are can be found at my first blog entry.

Case 3 - Relocate a PDB on the fly:

There are different use cases for relocating a pdb on the fly, the most common are to move a pdb to another hardware or to the/another cloud infrastructure.
As long as the "copy" is not opened read-write, the old PDB stays where it is. So you can, e.g., create a remote_listener entry for the pdb at the new environment and let it point to the listener at the environment you want to move from. If you do so, the remote listener (at the original location) will work as some kind of proxy to the listener at the new environment. The new PDB then can be used, e.g. by an application server, without any changes at the tns/jdbc connect. Be aware, that the service name must be the same on both sides for the relocated PDB and that you need to change the application connect to the new location somewhere later. If you forget this and you start decommissioning the old environment, the application and users can not work with the moved pdb.

How do you have moved a database with the "good old times" tools? Export and import? Well, this means you get a nice, reorganised database, but this isn't a fast approach if you look at downtime. Creating a data guard environment or using any replication tools is sometimes not possible due to license issues or costs (Standard Edition, Golden Gate, Quest Shareplex, ...). Maybe you can clone the database using rman and recover the new environment manually, but this is a lot work and you may make a small mistake somewhere and the environment isn't recoverable. So the easiest and last chance is to shutdown the production and move everything (using compressing tools or maybe not) to the new environment. And the copy takes time and all the time the database is down...

What would you say to the following solution in short:
- Create a database link between two databases
- Enter one statement to move the database in the backgroud and let it be mounted (so you can work with the original production).
- Open the database at the new location (which means it is a last time synchronized with the original PDB and after that the original PDB is deleted automatically)

Sounds easys? IT IS that easy!

What are the steps you need to do in detail?

1. Create a common user and grant him rights (SYSOPER!), (let) open the PDB you want to move to another host/to the cloud
2. Create a database link to the CDB$ROOT (!) you want to move from the CDB$ROOT you want the PDB moved to.
3. Create the pdb with RELOCATE at the second CDB.
4. Open the new PDB read write (this step synchronizes the PDBs and drops the old one afterwards).

Script to run, connects first to the source, prepares the user, second to the target CDB, prepares the database link and relocates the database.

Again one last comment: I didn't tested the scripts as I have thrown things out. If something does not work, please come back and leave a comment, so I can fix it.
Thanks a lot!

Single Tenant ist unnütze Komplexität? / Single Tenant is only unnecessary complexity? - Remote snapshot clone

Some people may wait for the second part of my three blog posts I am writing about Single Tenancy and why (some / a lot / most) people do think it is unnecessary complexity or - even worser - it is unnecessary at all.

What you do need to use the scripts as they are can be found at my first blog entry.

Case 2 - Remote snapshot clone of a PDB:

Snapshoting databases using a database link does have a lot of use cases.
The following two of them I do like most:
"Fast Provisioning", e.g. of nightly builds to testers or developers and application upgrades.

For remote snapshot pdb clones you either use file systems with snapshot technology included (like ACFS or ZFS) or you use traditional file systems (then you need to set CLONE_DB=true at your spfile and restart the database).

Well, how do you provision a nightly build today in 5 minutes to your testers and developers? On 40 machines? RMAN clone? Datapump export and import? What about the disk space you need for that? Like to spend money on it? Then tell me which shares I should buy before you throw the money out of the window. 😉

Or how do you make an application upgrade? With a non CDB database you can use offline backup the database, online backup but note down the last SCN, configure flashback database, ... And how long does it need to return to the before-upgrade-image if something goes wrong with the application upgrade?

Guys, years ago I was an app consultant and I had to go back with an installation of an application three times (at the same customer), because the business people did find some issues they haven't seen (tested) on the test system before. It took me hours and a lot nerves to restore everything to the right point-in-time and to setup the standby environments later on again...And if you do this three times in 4 months... 😟

So, the trick you will use for all of these uses cases is in the future - remote snapshot clone. Fast, reliable, you don't need to touch the "production", because all changes are done in your "new" snapshoted PDB.

What are the steps you need to do?

1. Create a common user and grant him rights, open the PDB you want to make a snapshot from read only (if you don't use a snapshot capable file system);
2. Create a database link to the PDB you want to clone from the CDB$ROOT you want the PDB cloned to.
3. Create the snapshot clone remote at the second CDB.
4. Open the new PDB read write
5. Make your application upgrade or develop or test
1. if it is sucessful / if you need to keep the PDB - "alter pluggable database <xyz> materialize;"
2. if it is not successful or before you create another nightly build - drop the new PDB and open the original source PDB read write again

Script 1 to run at the source CDB$ROOT container prepares the source PDB to be snapshoted.
Script 2 to run at the target CDB$ROOT container where the snapshot clone should be created

Some notes to point 6: If you have dropped the snapshot clone PDB the source PDB can't be open read write. The error message you get is not telling you the truth, because the PDB files are on OS level also set to READ. So what you do need to do is to change the permission on OS level for the database files. So the right command will look like

sudo chmod 640 /u01/app/oracle/oradata/XFEE1/77F0ADE89C820516E0534738A8C0802B/datafile/*

After you have made these changes, the source PDB can be opened read write again. It's part of the script 2, but as the folder names are different on different databases, I have made a comment out of it.

One last comment: I didn't tested the scripts as I have thrown things out. If something does not work, please come back and leave a comment, so I can fix it.
Thanks a lot!


By the way - happy New Year 2019 everyone!