Sunday, January 3, 2016

Oracle Clusterware - Startup (Internals)

 It's an long standing post for me to share how the CRS start-up works. Hope every one known with pictorial representation of CRS start-up provided in oracle documentation ( ) and each different process how it works. Perhaps I was intended to share how the framework and inter process communication works over agent framework. Agent uses below 4 internal threaded modules for communication across the start-up and live ness of cluster.


Apart from above modules, the communication over IPC gets differs for different modules (its like logical entity). Example ftok function get used from unix flavor perspective to get unique id for respective identified. When we try to use ftok across different threads they try to get same unique identifier and able to same information accessing/sharing across different threads

Example :

The reason for the system scope is that you want two or more independent processes to have access to the same IPC resources. So if you have two programs, both of which execute key = ftok("/home/oracle/somefile", 'b');, both will get the same token and can therefor access the same resources (semaphores, shared memory, message queues). That's the whole point of Inter Process Communication.

I have shared the Cluster ware time based startup Internals - which would show how cluster communications against different modules.

Hope this will answer most of the DBA's questions now and then , how cluster startup works (interview question)

Hope it helps.

Saturday, December 26, 2015

TNS-Linux Error: 29: Illegal seek

I will discuss the TNS-Linux Error, as the name suggest we have two parts over this error. Initial part is TNS + Linux, the combination makes the network and Oracle net8 (network) to work to-gather on initial startup. Oracle LSB (module/program is going - lsnrctl) make a check with type of protocol and initial port is free/available for use. During this process, we may came across the issues with socket binding with port and protocol mis-match (access issues).  I would be sharing the demo with simple typo (typing mistake of word PROTOCOL) raised the below issue (snapshot error) and it took around 45 mins to debug on a fresh day, on start-up of server.

[oracle@standby admin]$ lsnrctl start LISTEN_STDBY

LSNRCTL for Linux: Version - Production on 26-DEC-2015 09:43:14

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Starting /u01/app/oracle/product/11.2.0/db_1/bin/tnslsnr: please wait...

TNS-12537: TNS:connection closed
 TNS-12560: TNS:protocol adapter error
  TNS-00507: Connection closed
   Linux Error: 29: Illegal seek

By enabling listener trace and strace - I was able to figure out, why protocol was getting broken as part of program logic (It hard some times to understand or get through with issue). I always referred to basic OSI model for my understanding and troubleshooting when such issues were experienced  - OSI Model

Example from socket and protocol bind

if((socketHandle = socket(AF_INET, SOCK_STREAM, IPPROTO_IP)) < 0)
AF_INET is for IPV4 and AV_INET6 for IPV6

Basic syntax : socket(int socket_family, int socket_type, int protocol); 

Once socket is open or available then it needs to bind to certain port
bind(): bind the socket to a local socket address. This assigns a name to the socket.
Further, its listen for clients messages or services (similarity, if we observe in the demo, strace logs, it failed at initial phase of socket connection (error out) and we can see brk() call further to next sequence.

Demonstration of Issue/ Fix at my practical blog :

Hope it's find useful.

Friday, December 25, 2015

Recover/Open Unknown Database (Consistent Files)

I would discuss about the recovery (opening) consistent database, when some one shared or provided the database files. I have seen people asking / looking for steps how to proceed when clients or some one shared the database files for creating new development / staging envionments. They usually tend to miss or doesn't provide any good information about database name or spfile / pfile backups. Even certain situations comes up when they are restoring pretty old backup datafiles and no one is aware of any information (perhaps they are sure of database is consistent).

Today, I will share the demo (a practical step by step approach) how to proceed, when certain situation arises (or secenario comes on to your plate).

Demo at my Oracle Practicals :

Feel free to come back with your queries, if you have any. Hope it helps for upcoming DBA's

Thursday, December 24, 2015

Duplicate Database On Same Host (parameter value convert)

Today, will go through the real use of parameter value convert parameter of rman, while performing the database duplicate on same host/server. DBA's often get confused with db_file_name_convert and  parameter_value_convert by just specifying the values of conversion of all initialization parameters.

When we set some parameters with help of parameter_value_convert, rman processes SET after PARAMETER_VALUE_CONVERT. If PARAMETER_VALUE_CONVERT sets the filename specified by a parameter, and if SET sets the filename specified by the same parameter, then the SET value overrides the PARAMETER_VALUE_CONVERT setting.

Consider the case where controlfiles are not mapped to db_file_name_convert paths but multiplexed to two different locations. During such cases we come across issue with ORA-19504: failed to create file and ORA-19849: error while reading errors.

It's an misconception, as they didn't understood the parameter use, by placing value conversions parameter_value_convert it would get handle the issue of controlfile multiplexing. That;s say purely we can skip the parameter control_file (location in auxiliary instance) and still the duplicate database works smoothly with out any issues.

Below note is from Oracle doc:

Note: If DB_FILE_NAME_CONVERT is specified on the DUPLICATE command, then its filename settings override competing settings specified by SPFILE SET.

One of the question/query raised in forums : (Second physical standby on the same host)

Rman is intelligent enough to set the parameters with alter system as per parameter_value_convert values.  Please find the demo

RMAN Backup copy (with reuse option)

Today, we will try to understand, how rman works in choosing the right backup (scn) while taking the backups (image copy). Image copy backup is generally from end-user perspective its an OS image copy as we carry out. Recovery manager provides wonderful option "reuse", which allows rman to reuse the existing copy (instead of taking/creating new files all to-gather). It's allows to save the space. As every one should make sure have enough space in FRA. Perhaps downside of this (some times), during the errors or abnormal termination of process, it would end-up in partially updated datafiles. During restore and recovery we wouldn't need the old and latest touched/modified files by rman. For better understanding I have provided a demo which would showcase how rman is choosing the minimum SCN where to start the recovery.

Same question was raised in forums :

Feel free to come back to with questions/queries and let me know what else you are interested with demos of troubleshooting.


Wednesday, July 10, 2013

Oracle 12c ..available for windows


Today, I was able to see the download available for windows. It’s provides another round of testing new features on windows environment too.


Enjoy the 12c and lot’s of features on your way for Database Systems.!!