[omniORB] timeout detection (client-side)

[Michael Kilburn]

Hi,

Is it possible to reliably distinguish these three conditions? :
- server can't be contacted (i.e. it dropped out of network and it
can't detect that, extremely overloaded, deadlocked, whatever) with
timeout set by corresponding function
- server process is dead (i.e. box is alive, but related port is
closed, box is switched off and network could detect that...)
- other conditions, such as errors in client's ORB, errors in server's
ORB (e.g. out of memory, system errors)

Currently I am totally confused by how omniORB behaves (in some cases
generates COMM_FAILURE, in some -- TRANSIENT)...

Also, what about providing nice extension -- automatically catch
std::exception in request dispatcher and generate some CORBA system
exception (passing exception::what() to the client). Right now, I have
to go through tedious process of adding this support for almost every
function in idl by catching and converting std::exception to my own
InternalError.

[Fischer, Clemens]

Hi,

COMM_FAILURE is raised when the server closes a connection while there are outstanding requests. This usually happens when the server crashes during execution of an interface method. When there are no outstanding requests, the server may close the connection at any time to protect itself agains too many open sockets. On the next request the client simply reconnects (transparently in the client-side ORB).

TRANSIENT is raised when the server can't be connected, i.e. the client's TCP/IP connect call fails. This happens when the server process isn't running or doesn't listen, or the network is degraded, etc.

Since omniORB does not support TIMEOUT exceptions out of the box, you have to set up a TRANSIENT exception handler to create them as a result of a TRANSIENT exception with a special minor code. This is described in chapter 4.7 of the omniORB User's Guide.


// TRANSIENT exception handler function
//
static CORBA::Boolean transient_exception_handler(
void*, CORBA::ULong, const CORBA::TRANSIENT& exc)
{
if (exc.minor() == omni::TRANSIENT_CallTimedout)
{
// convert TRANSIENT to TIMEOUT exception
throw CORBA::TIMEOUT(exc.minor(), exc.completed());
}

return 0;
}

// set client call timeout for one object
//
CORBA::ULong timeout = 1000; // ms
omniORB::setClientCallTimeout(obj, timeout);
omniORB::installTransientExceptionHandler(
obj, 0, transient_exception_handler);

// or set client call timeout globally
//
CORBA::ULong timeout = 1000; // ms
omniORB::setClientCallTimeout(timeout);
omniORB::installTransientExceptionHandler(
0, transient_exception_handler);


-------
Clemens




-----Urspr?ngliche Nachricht-----
Von: omniorb-list-***@omniorb-support.com [mailto:omniorb-list-***@omniorb-support.com] Im Auftrag von Michael Kilburn
Gesendet: Dienstag, 20. November 2007 11:42
An: omniorb-***@omniorb-support.com
Betreff: [omniORB] timeout detection (client-side)

Hi,

Is it possible to reliably distinguish these three conditions? :
- server can't be contacted (i.e. it dropped out of network and it
can't detect that, extremely overloaded, deadlocked, whatever) with
timeout set by corresponding function
- server process is dead (i.e. box is alive, but related port is
closed, box is switched off and network could detect that...)
- other conditions, such as errors in client's ORB, errors in server's
ORB (e.g. out of memory, system errors)

Currently I am totally confused by how omniORB behaves (in some cases
generates COMM_FAILURE, in some -- TRANSIENT)...

Also, what about providing nice extension -- automatically catch
std::exception in request dispatcher and generate some CORBA system
exception (passing exception::what() to the client). Right now, I have
to go through tedious process of adding this support for almost every
function in idl by catching and converting std::exception to my own
InternalError.

[Michael Kilburn]

Hi,

Hmm... From my tests I have a feeling that COMM_FAILURE gets generated
if underlying ORB had non-closed socket to the server (which died) and
and I attempt to send a request there. Which is kind of unpredictable,
since you never know if ORB has connection or not...
Question: in such cases, will omniORB (after detecting that socket
connection is dead) attempt to reestablish connection in the same call
or I need to catch these cases and call again, expecting TRANSIENT?


My original intention was to find reliably whether:
- server is 100% dead (i.e. box is reachable, but related port is not listening)
- server is potentially alive (i.e. socket can't connect due to
timeout (due to network problems, or server overload))
- server is 100% alive, but has problems in business logic (i.e.
request was sent and received successfully, but request processing
timed out, e.g. due to deadlock)
- server is 100% alive, but there are problems in CORBA layer (no
resources, protocol incompatibilities and so on)

Is it possible to do that?


On Nov 20, 2007 10:21 PM, Fischer, Clemens

Post by Fischer, Clemens
Hi,
COMM_FAILURE is raised when the server closes a connection while there are outstanding requests. This usually happens when the server crashes during execution of an interface method. When there are no outstanding requests, the server may close the connection at any time to protect itself agains too many open sockets. On the next request the client simply reconnects (transparently in the client-side ORB).
TRANSIENT is raised when the server can't be connected, i.e. the client's TCP/IP connect call fails. This happens when the server process isn't running or doesn't listen, or the network is degraded, etc.
Since omniORB does not support TIMEOUT exceptions out of the box, you have to set up a TRANSIENT exception handler to create them as a result of a TRANSIENT exception with a special minor code. This is described in chapter 4.7 of the omniORB User's Guide.
// TRANSIENT exception handler function
//
static CORBA::Boolean transient_exception_handler(
void*, CORBA::ULong, const CORBA::TRANSIENT& exc)
{
if (exc.minor() == omni::TRANSIENT_CallTimedout)
{
// convert TRANSIENT to TIMEOUT exception
throw CORBA::TIMEOUT(exc.minor(), exc.completed());
}
return 0;
}
// set client call timeout for one object
//
CORBA::ULong timeout = 1000; // ms
omniORB::setClientCallTimeout(obj, timeout);
omniORB::installTransientExceptionHandler(
obj, 0, transient_exception_handler);
// or set client call timeout globally
//
CORBA::ULong timeout = 1000; // ms
omniORB::setClientCallTimeout(timeout);
omniORB::installTransientExceptionHandler(
0, transient_exception_handler);

[Fischer, Clemens]

-----Urspr?ngliche Nachricht-----
Von: Fischer, Clemens
Gesendet: Mittwoch, 21. November 2007 14:01
An: 'Michael Kilburn'
Betreff: AW: [omniORB] timeout detection (client-side)

Hi,
when the server died and thus closed the connected while having outstanding requests, COMM_FAILURE must be raised. From the client's point of view the situation is the same as if the server had deliberately closed the connection.
As far is i know (but i don't know omniORB's internals very well) there is no reconnection behind the scenes in such a situation. Your server doesn't behave well, so the client should be informed about.


-----Urspr?ngliche Nachricht-----
Von: Michael Kilburn [mailto:***@gmail.com]
Gesendet: Mittwoch, 21. November 2007 13:52
An: Fischer, Clemens
Cc: omniorb-***@omniorb-support.com
Betreff: Re: [omniORB] timeout detection (client-side)

Hi,

Hmm... From my tests I have a feeling that COMM_FAILURE gets generated
if underlying ORB had non-closed socket to the server (which died) and
and I attempt to send a request there. Which is kind of unpredictable,
since you never know if ORB has connection or not...
Question: in such cases, will omniORB (after detecting that socket
connection is dead) attempt to reestablish connection in the same call
or I need to catch these cases and call again, expecting TRANSIENT?


My original intention was to find reliably whether:
- server is 100% dead (i.e. box is reachable, but related port is not listening)
- server is potentially alive (i.e. socket can't connect due to
timeout (due to network problems, or server overload))
- server is 100% alive, but has problems in business logic (i.e.
request was sent and received successfully, but request processing
timed out, e.g. due to deadlock)
- server is 100% alive, but there are problems in CORBA layer (no
resources, protocol incompatibilities and so on)

Is it possible to do that?


On Nov 20, 2007 10:21 PM, Fischer, Clemens

Post by Fischer, Clemens
Hi,
COMM_FAILURE is raised when the server closes a connection while there are outstanding requests. This usually happens when the server crashes during execution of an interface method. When there are no outstanding requests, the server may close the connection at any time to protect itself agains too many open sockets. On the next request the client simply reconnects (transparently in the client-side ORB).
TRANSIENT is raised when the server can't be connected, i.e. the client's TCP/IP connect call fails. This happens when the server process isn't running or doesn't listen, or the network is degraded, etc.
Since omniORB does not support TIMEOUT exceptions out of the box, you have to set up a TRANSIENT exception handler to create them as a result of a TRANSIENT exception with a special minor code. This is described in chapter 4.7 of the omniORB User's Guide.
// TRANSIENT exception handler function
//
static CORBA::Boolean transient_exception_handler(
void*, CORBA::ULong, const CORBA::TRANSIENT& exc)
{
if (exc.minor() == omni::TRANSIENT_CallTimedout)
{
// convert TRANSIENT to TIMEOUT exception
throw CORBA::TIMEOUT(exc.minor(), exc.completed());
}
return 0;
}
// set client call timeout for one object
//
CORBA::ULong timeout = 1000; // ms
omniORB::setClientCallTimeout(obj, timeout);
omniORB::installTransientExceptionHandler(
obj, 0, transient_exception_handler);
// or set client call timeout globally
//
CORBA::ULong timeout = 1000; // ms
omniORB::setClientCallTimeout(timeout);
omniORB::installTransientExceptionHandler(
0, transient_exception_handler);

[Duncan Grisby]

Post by Michael Kilburn
Hmm... From my tests I have a feeling that COMM_FAILURE gets generated
if underlying ORB had non-closed socket to the server (which died) and
and I attempt to send a request there. Which is kind of unpredictable,
since you never know if ORB has connection or not...
Question: in such cases, will omniORB (after detecting that socket
connection is dead) attempt to reestablish connection in the same call
or I need to catch these cases and call again, expecting TRANSIENT?

If a connection has previously been used successfully, and a subsequent
call fails when marshalling the request, omniORB will transparently try
to reconnect, and you will see a TRANSIENT. If the connection breaks in
the middle of sending a request (e.g. because the server crashes in
response to the call), you see COMM_FAILURE.

Post by Michael Kilburn
- server is 100% dead (i.e. box is reachable, but related port is not
listening)

In that case, you will definitely get TRANSIENT with minor code
TRANSIENT_ConnectFailed.

Post by Michael Kilburn
- server is potentially alive (i.e. socket can't connect due to
timeout (due to network problems, or server overload))

If the server is too busy to accept the new connection, you will still
see TRANSIENT_ConnectFailed, since the client can't tell the
difference. If it accepts the connection but is too busy to start
processing it, you'll get TRANSIENT_CallTimedOut.

Post by Michael Kilburn
- server is 100% alive, but has problems in business logic (i.e.
request was sent and received successfully, but request processing
timed out, e.g. due to deadlock)

In that case, you'll get TRANSIENT with minor code
TRANSIENT_CallTimedOut. There is no way a client can tell the difference
between this situation and the case that the server is just too busy to
respond.

Post by Michael Kilburn
- server is 100% alive, but there are problems in CORBA layer (no
resources, protocol incompatibilities and so on)

You'll most likely get some other system exception like MARSHAL,
BAD_PARAM, BAD_OPERATION, etc. If the server gets confused in a way that
causes it to drop the connection, you'll get a COMM_FAILURE.

[Michael Kilburn]

Post by Duncan Grisby

Post by Michael Kilburn
- server is potentially alive (i.e. socket can't connect due to
timeout (due to network problems, or server overload))

If the server is too busy to accept the new connection, you will still
see TRANSIENT_ConnectFailed, since the client can't tell the
difference. If it accepts the connection but is too busy to start
processing it, you'll get TRANSIENT_CallTimedOut.

What will happen if client set a timeout value (2 seconds) and server
is too busy to accept a connection. Will client:
- timeout according to underlying protocol settings (usually 20 sec
(?) for TCP/IP) and get TRANSIENT_ConnectFailed
- or it will return after 2 sec with TRANSIENT_CallTimedOut (or something else?)

i.e. does omniORB::setClientCallTimeout() guarantee that we'll spend
at most X seconds in the call in every possible scenario?

[Duncan Grisby]

Post by Michael Kilburn
What will happen if client set a timeout value (2 seconds) and server
- timeout according to underlying protocol settings (usually 20 sec
(?) for TCP/IP) and get TRANSIENT_ConnectFailed
- or it will return after 2 sec with TRANSIENT_CallTimedOut (or
something else?)

It will time out after 2 seconds with TRANSIENT_ConnectFailed.

Post by Michael Kilburn
i.e. does omniORB::setClientCallTimeout() guarantee that we'll spend
at most X seconds in the call in every possible scenario?

Yes, it will guarantee that, to within the limits of scheduling issues
and such. omniORB doesn't try to make hard real-time guarantees, but it
does make a soft guarantee to always time out after the timeout period.

Cheers,

Duncan.