[omniORB] performance limits of omniNames (COMM_FAILURE)

[Jaromír Talíř]

Hello,

we are using omniORB as a communication platform in just created
domain registry system for .cz TLD (about 350000 domains, released as
open source on http://fred.nic.cz). Both python and c++ parts are really
good. Many thanks to authors!

But recently we had noticed on our production server (omniorb-4.0.6),
that in high load, omniNames started refuse to answer 'resolve' command
throwing CORBA/COMM_FAILURE. I managed to simulate this situation (hope
it's the same situation) with omniorb-4.1.0 by this simple script:

---- begin of test.sh -----
#!/bin/bash
while /bin/true
do
nameclt -ior corbaname::localhost resolve Name.Object > /dev/null
done
---- end of test ----

for i in `seq 1 300`; do { ./test.sh & }; done

After few minutes it start output:

Caught a TRANSIENT exception when trying to validate the type of the
NamingContext. Is the naming service running?
Caught a TRANSIENT exception when trying to validate the type of the
NamingContext. Is the naming service running?
...

And after small time:

Unexpected CORBA COMM_FAILURE exception when trying to narrow the
NamingContext.
Unexpected CORBA COMM_FAILURE exception when trying to narrow the
NamingContext.
Unexpected CORBA COMM_FAILURE exception when trying to narrow the
NamingContext.
Unexpected CORBA COMM_FAILURE exception when trying to narrow the
NamingContext.
...

What should it be? I tried to lookup omniORB thread options and I think
defaults are fine. I expect that when no corba resources (threads?) are
available, call is blocked waiting for resource(clientCallTimeOutPeriod
is 0 by default), so there shouldn't be any failure (and only efficiancy
will drop), is it true?

Regards

Jaromir Talir
CZ.NIC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3870 bytes
Desc: not available
Url : http://www.omniorb-support.com/pipermail/omniorb-list/attachments/20071107/65886da9/smime.bin

[Sean Parker]

Hello -

I had some heavy-throughput issues also and I was getting
COMM_FAILURES due to connection limitations - I needed to
increase allowed files at the OS level, and I played with
the threading/connection options as well to get things to
flow. (Since my code is now at a "previous employer" I'm
afraid I can't just look up exactly my solutions)

I recall it wasn't so much an OS-level thread allowance
as much as it was a file allowance. If on Linux, look into
sysctl issues for increasing the allowed files.

I had great luck with throughput but didn't document
clearly the exact steps I needed to do to get it working
[to the point that I needed it to]. Once you iron this out,
can you provide feedback to authors so that this
information gets into the documentation? I think this is a
critical issue (the publication of resolving issues like
this) since:
1) it communicates that omniORB is being used in
heavy-duty computing environments,
2) it communicates that the users and developer actually
know what they're doing (_I_ don't doubt it!) and,
3) it helps others develop faster and with more
confidence in the design stage.

Cheers
Sean

Post by Jaromír Talíř
Hello,
we are using omniORB as a communication platform in
just created
domain registry system for .cz TLD (about 350000 domains,
released as
open source on http://fred.nic.cz). Both python and c++
parts are really
good. Many thanks to authors!
But recently we had noticed on our production server
(omniorb-4.0.6),
that in high load, omniNames started refuse to answer
'resolve' command
throwing CORBA/COMM_FAILURE. I managed to simulate this
situation (hope
it's the same situation) with omniorb-4.1.0 by this
---- begin of test.sh -----
#!/bin/bash
while /bin/true
do
nameclt -ior corbaname::localhost resolve Name.Object

Post by Jaromír Talíř
/dev/null

done
---- end of test ----

Post by Jaromír Talíř
for i in `seq 1 300`; do { ./test.sh & }; done

Caught a TRANSIENT exception when trying to validate the
type of the
NamingContext. Is the naming service running?
Caught a TRANSIENT exception when trying to validate the
type of the
NamingContext. Is the naming service running?
...
Unexpected CORBA COMM_FAILURE exception when trying to
narrow the
NamingContext.
Unexpected CORBA COMM_FAILURE exception when trying to
narrow the
NamingContext.
Unexpected CORBA COMM_FAILURE exception when trying to
narrow the
NamingContext.
Unexpected CORBA COMM_FAILURE exception when trying to
narrow the
NamingContext.
...
What should it be? I tried to lookup omniORB thread
options and I think
defaults are fine. I expect that when no corba resources
(threads?) are
available, call is blocked waiting for
resource(clientCallTimeOutPeriod
is 0 by default), so there shouldn't be any failure (and
only efficiancy
will drop), is it true?
Regards
Jaromir Talir
CZ.NIC

Post by Jaromír Talíř
_______________________________________________

omniORB-list mailing list

http://www.omniorb-support.com/mailman/listinfo/omniorb-list
God Bless
Sean Parker




__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Duncan Grisby]

Post by Jaromír Talíř
we are using omniORB as a communication platform in just created
domain registry system for .cz TLD (about 350000 domains, released as
open source on http://fred.nic.cz). Both python and c++ parts are really
good. Many thanks to authors!
But recently we had noticed on our production server (omniorb-4.0.6),
that in high load, omniNames started refuse to answer 'resolve' command
throwing CORBA/COMM_FAILURE. I managed to simulate this situation (hope

There are two scalability issues here. One is omniORB scalability. Its
default policies are to use one thread per connection, and to keep idle
connections open for quite a while. That means it can hit the OS's limit
on open file descriptors or on running threads quite quickly. To help
with that, you can set omniORB to thread pool mode (with quite a large
pool size), and set the inConScanPeriod lower so idle connections are
closed much more quickly. You can also make sure the process has as high
a limit of file descriptors as the OS will allow.

The other issue is scalability of omniNames itself. It's a really simple
thing that isn't intended to scale much. In particular, if you store a
lot of names in a flat structure, it's dreadful because it uses a linear
scan through linked lists of names when looking things up. If you need
to store lots of name bindings, you should come up with some kind of
hierarchy to avoid too much linear behaviour.

What are you using omniNames for? It might well be better for you to
use something else to store object references.

Cheers,

Duncan.

[Sean Parker]

Excellent points Duncan -

I'd suggest to people that if it's that much of a
big-deal to "pound omniNames beyond it's intended usage"
than the developers can take different approaches to the
problem:
1) copy/rename omniNames itself, implementing a binary
tree name lookup if that will suit their needs;
2) implement some sort of
redundancy/fail-over/load-balancing policy and have
multiple omniNames instances in there system.

"2" brings up an interesting point - is there an OMG
standard, or other intended/built-in mechanism for handling
redundancy in CORBA?

This issue seems rather basic, and something all systems
could use. It's not so much another "service" (like Time or
Trading) but is more involved in the nameService/initial
references realm of things.

For example, I had designed such a scheme in the system
we had in my previous job, but never was able to implement
it. It basically provided for optional "redundancy
policies" for an application as client, and for the server.
For example a client would want the "SystemConfiguration"
(SC) reference, and if there werr two SC instances in the
system, a primary and a secondary on two servers, then a
reference manager in the application would know, based on
command-line policy declarations, whether to return the
"primary SC" or "secondary SC" (because they would have
been registered in the name service in a "primary" or
"secondary" name context), without the client code
designating/caring which was returned. Is there anything
else like that out there [in a standardized form]?

This issue of nameService performance could be
solved/mitigated with such a mechanism, and then
scalability of any part of a CORBA system becomes almost a
mute point. Any comments?

Cheers and God Bless
Sean

On Wednesday 7 November,

Post by Jaromír Talíř
we are using omniORB as a communication platform in

just created

Post by Jaromír Talíř
domain registry system for .cz TLD (about 350000

domains, released as

Post by Jaromír Talíř
open source on http://fred.nic.cz). Both python and c++

parts are really

Post by Jaromír Talíř
good. Many thanks to authors!
But recently we had noticed on our production server

(omniorb-4.0.6),

Post by Jaromír Talíř
that in high load, omniNames started refuse to answer

'resolve' command

Post by Jaromír Talíř
throwing CORBA/COMM_FAILURE. I managed to simulate this

situation (hope

Post by Jaromír Talíř
it's the same situation) with omniorb-4.1.0 by this

There are two scalability issues here. One is omniORB
scalability. Its
default policies are to use one thread per connection,
and to keep idle
connections open for quite a while. That means it can hit
the OS's limit
on open file descriptors or on running threads quite
quickly. To help
with that, you can set omniORB to thread pool mode (with
quite a large
pool size), and set the inConScanPeriod lower so idle
connections are
closed much more quickly. You can also make sure the
process has as high
a limit of file descriptors as the OS will allow.
The other issue is scalability of omniNames itself. It's
a really simple
thing that isn't intended to scale much. In particular,
if you store a
lot of names in a flat structure, it's dreadful because
it uses a linear
scan through linked lists of names when looking things
up. If you need
to store lots of name bindings, you should come up with
some kind of
hierarchy to avoid too much linear behaviour.
What are you using omniNames for? It might well be
better for you to
use something else to store object references.
Cheers,
Duncan.
--
-- Duncan Grisby --
-- http://www.grisby.org --
_______________________________________________
omniORB-list mailing list

http://www.omniorb-support.com/mailman/listinfo/omniorb-list
God Bless
Sean Parker




__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com