[omniORB] Why do oneway requests hang on disconnected network?

[Tuyen Chau]

Why do "oneway" requests hang, instead of return an COM_FAILURE
exception, when the network is disconnected? As part of testing our
product, we unplugged the network cable. We were surprised to find that
these oneway requests executed without errors for a good 5-10 minutes or
so, then they blocked indefinitely. If we replaced the network cable,
the calls eventually unblocked and everything worked again. Our best
guess at the moment is that there is a data buffer for outgoing requests
and the oneway requests block when the buffer is full.

Is there any way to alter this behavior and receive a COM_FAILURE
exception instead?

Our oneway requests are being used as "callbacks" when the server needs
to send events to its clients to notify them of changes in the system.

Best Regards,
Tuyen

[Luke Deller]

Hi Tuyen,

I suspect that this behaviour is not caused by omniORB but rather by the
underlying TCP/IP transport. The data buffer for outgoing requests that
you mention is probably the TCP socket's send buffer.

Which OS are you using? Can you configure it to shut down the network
interface when it senses that the network cable has been unplugged?

The CORBA spec does state that oneway calls are not guaranteed to be
delivered. To address the problem of indefinitely blocking, have you
tried setting a client-side timeout on the oneway invocation?

http://omniorb.sourceforge.net/omni40/omniORB/omniORB008.html#sec:timeou
tAPI

Regards,
Luke.

-----Original Message-----
From: omniorb-list-***@omniorb-support.com
[mailto:omniorb-list-***@omniorb-support.com] On Behalf Of Tuyen
Chau
Sent: Thursday, 16 November 2006 11:17 AM
To: OmniOrb
Subject: [SPAM_HeaderCheck] - [omniORB] Why do oneway requests hang on
disconnected network? - Email has different SMTP TO: and MIME TO: fields
in the email addresses

Why do "oneway" requests hang, instead of return an COM_FAILURE
exception, when the network is disconnected? As part of testing our
product, we unplugged the network cable. We were surprised to find that

these oneway requests executed without errors for a good 5-10 minutes or

so, then they blocked indefinitely. If we replaced the network cable,
the calls eventually unblocked and everything worked again. Our best
guess at the moment is that there is a data buffer for outgoing requests

and the oneway requests block when the buffer is full.

Is there any way to alter this behavior and receive a COM_FAILURE
exception instead?

Our oneway requests are being used as "callbacks" when the server needs
to send events to its clients to notify them of changes in the system.

Best Regards,
Tuyen


**********************************************************************************************

Important Note
This email (including any attachments) contains information which is
confidential and may be subject to legal privilege. If you are not
the intended recipient you must not use, distribute or copy this
email. If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this
email are not necessarily the views of XPlan Technology.

It is the duty of the recipient to virus scan and otherwise test the
information provided before loading onto any computer system.
Xplan Technology does not warrant that the
information is free of a virus or any other defect or error.
**********************************************************************************************

[Wernke zur Borg]

I would have expected that oneway calls always return immediately,
irrespective of the state of any underlying transport, and that they are
also independent of client-side timeout settings. That is also what I
have experienced so far, and I have not had any blocking problems. This
is on Solaris 8 with omniORB 4.0.6. I am curious about the answer to
this question.

Regards, Wernke

Post by Luke Deller
Hi Tuyen,
I suspect that this behaviour is not caused by omniORB but
rather by the
underlying TCP/IP transport. The data buffer for outgoing
requests that
you mention is probably the TCP socket's send buffer.
Which OS are you using? Can you configure it to shut down the network
interface when it senses that the network cable has been unplugged?
The CORBA spec does state that oneway calls are not guaranteed to be
delivered. To address the problem of indefinitely blocking, have you
tried setting a client-side timeout on the oneway invocation?
http://omniorb.sourceforge.net/omni40/omniORB/omniORB008.html#
sec:timeou
tAPI
Regards,
Luke.
-----Original Message-----
Chau
Sent: Thursday, 16 November 2006 11:17 AM
To: OmniOrb
Subject: [SPAM_HeaderCheck] - [omniORB] Why do oneway requests hang on
disconnected network? - Email has different SMTP TO: and MIME
TO: fields
in the email addresses
Why do "oneway" requests hang, instead of return an COM_FAILURE
exception, when the network is disconnected? As part of testing our
product, we unplugged the network cable. We were surprised
to find that
these oneway requests executed without errors for a good 5-10
minutes or
so, then they blocked indefinitely. If we replaced the
network cable,
the calls eventually unblocked and everything worked again. Our best
guess at the moment is that there is a data buffer for
outgoing requests
and the oneway requests block when the buffer is full.
Is there any way to alter this behavior and receive a COM_FAILURE
exception instead?
Our oneway requests are being used as "callbacks" when the
server needs
to send events to its clients to notify them of changes in the system.
Best Regards,
Tuyen
**************************************************************
********************************
Important Note
This email (including any attachments) contains information which is
confidential and may be subject to legal privilege. If you are not
the intended recipient you must not use, distribute or copy this
email. If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this
email are not necessarily the views of XPlan Technology.
It is the duty of the recipient to virus scan and otherwise test the
information provided before loading onto any computer system.
Xplan Technology does not warrant that the
information is free of a virus or any other defect or error.
**************************************************************
********************************
_______________________________________________
omniORB-list mailing list
http://www.omniorb-support.com/mailman/listinfo/omniorb-list

[Duncan Grisby]

Post by Tuyen Chau
Why do "oneway" requests hang, instead of return an COM_FAILURE
exception, when the network is disconnected? As part of testing our
product, we unplugged the network cable. We were surprised to find
that these oneway requests executed without errors for a good 5-10
minutes or so, then they blocked indefinitely. If we replaced the
network cable, the calls eventually unblocked and everything worked
again. Our best guess at the moment is that there is a data buffer
for outgoing requests and the oneway requests block when the buffer is
full.

omniORB doesn't buffer requests at all. In a oneway request, it simply
sends the data through the TCP socket and carries on its way. The
buffering you are seeing is in the TCP stack. Eventually, if the server
isn't responding (because the cable's not there), the TCP stack will
block when omniORB tries to send.

If the OS doesn't notice that the connection is broken, it won't tell
omniORB when omniORB tries to send, which is why you see that you can
send lots of oneway requests before anything untoward happens. The way
TCP works, there's no way to tell that a cable has been unplugged and
quickly close the connection.

Post by Tuyen Chau
Is there any way to alter this behavior and receive a COM_FAILURE
exception instead?

If you set a timeout on the calls, they will timeout if the send call
blocks, leading to a COMM_FAILURE exception. That won't make it fail any
quicker, though, because the send won't block and therefore timeout
until the TCP buffers are full.

The only other alternative is to modify omniORB so it sets the
SO_KEEPALIVE socket option on its tcp sockets. That way the OS will send
keepalive packets, and tear down the connection if the keepalives are
lost. But with that, you're at the mercy of the OS as to when it starts
sending keepalives, and once it does, how often it sends them and how
many must go missing before it gives up. See this from the Linux tcp
manpage for example:

SYSCTLS
These variables can be accessed by the /proc/sys/net/ipv4/* files or
with the sysctl(2) interface. In addition, most IP sysctls also apply
to TCP; see ip(7).
...
tcp_keepalive_intvl
The number of seconds between TCP keep-alive probes. The
default value is 75 seconds.

tcp_keepalive_probes
The maximum number of TCP keep-alive probes to send before giv-
ing up and killing the connection if no response is obtained
from the other end. The default value is 9.

tcp_keepalive_time
The number of seconds a connection needs to be idle before TCP
begins sending out keep-alive probes. Keep-alives are only
sent when the SO_KEEPALIVE socket option is enabled. The
default value is 7200 seconds (2 hours). An idle connection is
terminated after approximately an additional 11 minutes (9
probes an interval of 75 seconds apart) when keep-alive is
enabled.


The default times mean that SO_KEEPALIVE is basically useless for your
situation unless you radically reduce the times, but the settings are
for the whole machine, not just your process.

Cheers,

Duncan.

[Sean Parker]

Any chance on implementing oneways as UDP messages? (This
would be ideal for setting up a work around in environments
"hostile" to "slow" protocols, although "unreliable").
Certainly, if I were to use oneway's, it'd only be with the
intent of optimizing some part of our system for speed -
something UDP would be a good fit for...
Of course that'd require work to the internals of OmniORB
to also listen on another port for possible incoming UDP
messages....

Sean Parker

Post by Tuyen Chau

Post by Tuyen Chau
Why do "oneway" requests hang, instead of return an

COM_FAILURE

Post by Tuyen Chau
exception, when the network is disconnected? As part

of testing our

Post by Tuyen Chau
product, we unplugged the network cable. We were

surprised to find

Post by Tuyen Chau
that these oneway requests executed without errors for

a good 5-10

Post by Tuyen Chau
minutes or so, then they blocked indefinitely. If we

replaced the

Post by Tuyen Chau
network cable, the calls eventually unblocked and

everything worked

Post by Tuyen Chau
again. Our best guess at the moment is that there is a

data buffer

Post by Tuyen Chau
for outgoing requests and the oneway requests block

when the buffer is

Post by Tuyen Chau
full.

omniORB doesn't buffer requests at all. In a oneway
request, it simply
sends the data through the TCP socket and carries on its
way. The
buffering you are seeing is in the TCP stack. Eventually,
if the server
isn't responding (because the cable's not there), the TCP
stack will
block when omniORB tries to send.
If the OS doesn't notice that the connection is broken,
it won't tell
omniORB when omniORB tries to send, which is why you see
that you can
send lots of oneway requests before anything untoward
happens. The way
TCP works, there's no way to tell that a cable has been
unplugged and
quickly close the connection.

Post by Tuyen Chau
Is there any way to alter this behavior and receive a

COM_FAILURE

Post by Tuyen Chau
exception instead?

If you set a timeout on the calls, they will timeout if
the send call
blocks, leading to a COMM_FAILURE exception. That won't
make it fail any
quicker, though, because the send won't block and
therefore timeout
until the TCP buffers are full.
The only other alternative is to modify omniORB so it
sets the
SO_KEEPALIVE socket option on its tcp sockets. That way
the OS will send
keepalive packets, and tear down the connection if the
keepalives are
lost. But with that, you're at the mercy of the OS as to
when it starts
sending keepalives, and once it does, how often it sends
them and how
many must go missing before it gives up. See this from
the Linux tcp
SYSCTLS
These variables can be accessed by the
/proc/sys/net/ipv4/* files or
with the sysctl(2) interface. In addition, most
IP sysctls also apply
to TCP; see ip(7).
...
tcp_keepalive_intvl
The number of seconds between TCP
keep-alive probes. The
default value is 75 seconds.
tcp_keepalive_probes
The maximum number of TCP keep-alive probes
to send before giv-
ing up and killing the connection if no
response is obtained
from the other end. The default value is
9.
tcp_keepalive_time
The number of seconds a connection needs to
be idle before TCP
begins sending out keep-alive probes.
Keep-alives are only
sent when the SO_KEEPALIVE socket option
is enabled. The
default value is 7200 seconds (2 hours).
An idle connection is
terminated after approximately an
additional 11 minutes (9
probes an interval of 75 seconds
apart) when keep-alive is
enabled.
The default times mean that SO_KEEPALIVE is basically
useless for your
situation unless you radically reduce the times, but the
settings are
for the whole machine, not just your process.
Cheers,
Duncan.
--
-- Duncan Grisby --
-- http://www.grisby.org --
_______________________________________________
omniORB-list mailing list

http://www.omniorb-support.com/mailman/listinfo/omniorb-list
God Bless
Sean Parker






____________________________________________________________________________________
Cheap talk?
Check out Yahoo! Messenger's low PC-to-Phone call rates.
http://voice.yahoo.com

[Duncan Grisby]

Any chance on implementing oneways as UDP messages? (This would be
ideal for setting up a work around in environments "hostile" to "slow"
protocols, although "unreliable"). Certainly, if I were to use
oneway's, it'd only be with the intent of optimizing some part of our
system for speed - something UDP would be a good fit for... Of course
that'd require work to the internals of OmniORB to also listen on
another port for possible incoming UDP messages....

It's not something mentioned in the CORBA specs, but there's no reason
it couldn't be added as an omniORB extension. It would probably be quite
a lot of work, since omniORB's transport system assumes a connection
oriented protocol.

If you want to work on it, let me know and I'll help you understand the
relevant bits of omniORB's internals.

Cheers,

Duncan.

[Sean Parker]

I thought that's what you'd say! :-)

In my spare seconds, I can take a look... let me know where
I should look first - for example: when a client app is
making a request on a stub, how does omniORB differentiate
between a oneway request (not needing a return value of
course) and one requiring a holding pattern waiting for a
reply?

Cheers
Sean

Post by Sean Parker
Any chance on implementing oneways as UDP messages?

(This would be

Post by Sean Parker
ideal for setting up a work around in environments

"hostile" to "slow"

Post by Sean Parker
protocols, although "unreliable"). Certainly, if I

were to use

Post by Sean Parker
oneway's, it'd only be with the intent of optimizing

some part of our

Post by Sean Parker
system for speed - something UDP would be a good fit

for... Of course

Post by Sean Parker
that'd require work to the internals of OmniORB to also

listen on

Post by Sean Parker
another port for possible incoming UDP messages....

It's not something mentioned in the CORBA specs, but
there's no reason
it couldn't be added as an omniORB extension. It would
probably be quite
a lot of work, since omniORB's transport system assumes a
connection
oriented protocol.
If you want to work on it, let me know and I'll help you
understand the
relevant bits of omniORB's internals.
Cheers,
Duncan.
--
-- Duncan Grisby --
-- http://www.grisby.org --

God Bless
Sean Parker






____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

[Duncan Grisby]

Post by Sean Parker
In my spare seconds, I can take a look... let me know where
I should look first - for example: when a client app is
making a request on a stub, how does omniORB differentiate
between a oneway request (not needing a return value of
course) and one requiring a holding pattern waiting for a
reply?

Whether a call is oneway or not is recorded in a callDescriptor object
-- see include/omniORB4/callDescriptor.h.

However, what you really need to understand first is how the connection
management works, which is far from trivial. You might want to start
reading src/lib/omniORB/orbcore/transport.txt, then start looking at the
implementations of the classes it mentions.

Cheers,

Duncan.