[omniORB] Network performance

[Default User]

I've been doing some test on Linux. I'd gathered some latency numbers
for a client/server message passing system, with both co-located on the
same box (localhost endpoint).

I then switched to having the client and server running on different
Linux boxes in our lab, connected via a simple high-speed LAN. The
performance went way down. Average latency for sending a message
increased by an order of magnitude.

This was certainly unexpected. A SOAP implementation that I was tested
showed almost no change in latency in similar circumstances. Is there
something I've set up incorrectly?



Brian


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Ridgway, Richard (London)]

Taking a punt, SOAP was an order of magnitude slower while co-located?
Why would you not expect the latency increase when going over the LAN?
Simple test below:

%ping localhost
PING ......... 56(84) bytes of data.
64 bytes from ......... icmp_seq=0 ttl=58 time=0.027 ms

%ping neighbouring-machine
PING .......... 56(84) bytes of data.
64 bytes from ............. icmp_seq=0 ttl=58 time=0.535 ms



-----Original Message-----
From: omniorb-list-***@omniorb-support.com
[mailto:omniorb-list-***@omniorb-support.com] On Behalf Of Default
User
Sent: 09 November 2007 00:24
To: omniorb-***@omniorb-support.com
Subject: [omniORB] Network performance


I've been doing some test on Linux. I'd gathered some latency numbers
for a client/server message passing system, with both co-located on the
same box (localhost endpoint).

I then switched to having the client and server running on different
Linux boxes in our lab, connected via a simple high-speed LAN. The
performance went way down. Average latency for sending a message
increased by an order of magnitude.

This was certainly unexpected. A SOAP implementation that I was tested
showed almost no change in latency in similar circumstances. Is there
something I've set up incorrectly?



Brian


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

_______________________________________________
omniORB-list mailing list
omniORB-***@omniorb-support.com
http://www.omniorb-support.com/mailman/listinfo/omniorb-list
--------------------------------------------------------

This message w/attachments (message) may be privileged, confidential or proprietary, and if you are not an intended recipient, please notify the sender, do not use or share it and delete it. Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Merrill Lynch. Subject to applicable law, Merrill Lynch may monitor, review and retain e-communications (EC) traveling through its networks/systems. The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or error-free. This message is subject to terms available at the following link: http://www.ml.com/e-communications_terms/. By messaging with Merrill Lynch you consent to the foregoing.
--------------------------------------------------------

[Richard Hirst]

Have a look at nerperf. See how fast that can bounce small packets
back and fore between the two machines in its RR tests (request-response).
That will give you an idea of what your hosts, NICs, and network are
capable of. Some NICs throttle the maximum interrupt rate, for example,
which can really limit throughput in a request-response type of test.

Richard

Post by Default User
I've been doing some test on Linux. I'd gathered some latency numbers
for a client/server message passing system, with both co-located on the
same box (localhost endpoint).
I then switched to having the client and server running on different
Linux boxes in our lab, connected via a simple high-speed LAN. The
performance went way down. Average latency for sending a message
increased by an order of magnitude.
This was certainly unexpected. A SOAP implementation that I was tested
showed almost no change in latency in similar circumstances. Is there
something I've set up incorrectly?
Brian
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
_______________________________________________
omniORB-list mailing list
http://www.omniorb-support.com/mailman/listinfo/omniorb-list

[Default User]

Post by Ridgway, Richard (London)
Taking a punt, SOAP was an order of magnitude slower while
co-located?

No, not in general. Depending on the type of message, SOAP took
three times as long or more in a round-trip, averaged over many
trials.

Post by Ridgway, Richard (London)
Why would you not expect the latency increase when going over the
LAN?

I did expect some, but thought that CORBA and SOAP would increase
at about the same rate.

Again, SOAP's performance was essentially the same. Omniorb's degraded
by an order of magnitude, so that box-to-box it's much worse than
gSOAP.

That doesn't seem right to me, so I'm looking to see if I've made
some error in set-up.

Post by Ridgway, Richard (London)
%ping localhost
PING ......... 56(84) bytes of data.
64 bytes from ......... icmp_seq=0 ttl=58 time=0.027 ms
%ping neighbouring-machine
PING .......... 56(84) bytes of data.
64 bytes from ............. icmp_seq=0 ttl=58 time=0.535 ms

I'll get those numbers.

Post by Ridgway, Richard (London)
-----Original Message-----
Default
User
Sent: 09 November 2007 00:24
Subject: [omniORB] Network performance
I've been doing some test on Linux. I'd gathered some latency numbers
for a client/server message passing system, with both co-located on
the
same box (localhost endpoint).
I then switched to having the client and server running on different
Linux boxes in our lab, connected via a simple high-speed LAN. The
performance went way down. Average latency for sending a message
increased by an order of magnitude.
This was certainly unexpected. A SOAP implementation that I was
tested
showed almost no change in latency in similar circumstances. Is there
something I've set up incorrectly?

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Default User]

I assume both cases (local and remote) were worse than omni but less
delta
between them?

Actually, no. The box-to-box tests had gSOAP beating omniorb. For the
localhost test, omniorb was somewhere around 3 to 10 times faster.
gSOAP being relatively unaffected (seemingly) by network latency,
it was then around 3 times faster to on par, depending on the
message type. The SOAP was notably slow with large arrays of doubles,
for instance, as we were using "document literal" mode.




Brian


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Scott Ross]

Something to remember about omniorb is that will use different transports if
they are available. By default it will use Unix domain sockets on the
localhost and TCP(or SSL) when going remote. It depends on the contents of
the IOR, the 'client/serverTransportRule' settings. That is likely to be
the biggest factor in the difference you see. There are other settings that
might affect the results -- calls/connection, connection/server,
threads/connection, POA policies, etc. But they (for me) have never made
'order of magnitude' differences.
enjoy
scott
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.omniorb-support.com/pipermail/omniorb-list/attachments/20071110/ced7c25e/attachment.htm

[Duncan Grisby]

Post by Default User

I assume both cases (local and remote) were worse than omni but less
delta between them?

Actually, no. The box-to-box tests had gSOAP beating omniorb. For the
localhost test, omniorb was somewhere around 3 to 10 times faster.
gSOAP being relatively unaffected (seemingly) by network latency,
it was then around 3 times faster to on par, depending on the
message type. The SOAP was notably slow with large arrays of doubles,
for instance, as we were using "document literal" mode.

I'd be very surprised if omniORB is slower than a SOAP implementation,
since SOAP just has a lot more work to do to generate and parse XML
messages, and the messages are much larger.

Can you post some more details about what your test is actually doing?
How much data are you sending in each call? What kind of data is it?
Are you just doing simple calls, or something more exotic with callbacks
or something?

What kind of absolute times do you get for your calls?

Cheers,

Duncan.

[Duncan Grisby]

Post by Duncan Grisby
What kind of absolute times do you get for your calls?

By the way, I did some quick tests on some not very fast hardware, doing
simple calls that sent and received an empty string. On a single
machine, with normal two-way calls, I got 20000 calls per second with
the TCP transport and 40000 calls per second with the Unix domain socket
transport. For oneway calls, where the effects of round-trip latency are
removed, I got 125000 calls per second with TCP and 175000 with Unix
sockets.

Between two machines, on a 100Mbit network, two-way calls were 7200
calls per second and oneways were 110000 per second.

How does that compare with your numbers?

Cheers,

Duncan.

[Default User]

Post by Duncan Grisby

Post by Default User

I assume both cases (local and remote) were worse than omni but

less

Post by Default User

delta between them?

Actually, no. The box-to-box tests had gSOAP beating omniorb.

I'd be very surprised if omniORB is slower than a SOAP
implementation,
since SOAP just has a lot more work to do to generate and parse XML
messages, and the messages are much larger.

I wasn't expecting it either, and in co-located tests Omniorb was
faster. I expected both to degrade about the same in the box-to-box
tests, with a bit of network latency added. That wasn't the case.
I need to present the results soon, and I want to make sure that
I have accurate findings.

Post by Duncan Grisby
Can you post some more details about what your test is actually
doing?
How much data are you sending in each call? What kind of data is it?
Are you just doing simple calls, or something more exotic with
callbacks or something?

Basically we devised IDL and WSDL documents that tried to be as
similar as possible. There were a variety of tests, most that
were just passing data via the parameters, with an integer return.

We had ones for passing arrays of integers or doubles, strings,
and a complex structure.

We'd then run in a tight loop calling the particular function
in the server repeatedly. The handler in the server didn't do
anything except immediately return. I can't give you any specifics
because I won't be back into the office until Monday.

Post by Duncan Grisby
What kind of absolute times do you get for your calls?

Off the top of my head (again the stuff is at work) it was coming out
to be something like .25 seconds to execute the 10000 calls for
small amounts of data when co-located. It would go to up 2.5 seconds
for the same data when going over the network.

As a data point, ping on "localhost" took about 10x as long as
well.

I can give you more specifics, like exact IDL and example code
either tomorrow or Monday.




__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Default User]

Post by Duncan Grisby

Post by Duncan Grisby
What kind of absolute times do you get for your calls?

By the way, I did some quick tests on some not very fast hardware,
doing
simple calls that sent and received an empty string. On a single
machine, with normal two-way calls, I got 20000 calls per second with
the TCP transport and 40000 calls per second with the Unix domain
socket
transport. For oneway calls, where the effects of round-trip latency
are
removed, I got 125000 calls per second with TCP and 175000 with Unix
sockets.
Between two machines, on a 100Mbit network, two-way calls were 7200
calls per second and oneways were 110000 per second.
How does that compare with your numbers?

I mentioned in an earlier message that 10000 calls were taking about
.25 seconds, so I'm seeing like 40000 per second. That drops to
about 4000 between boxes. I've only used TCP as transport. I specify
the endpoint in all cases for both client and server, either
-ORBendPoint giop:tcp:localhost:12345 or the remote IP address.


Brian


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Default User]

Post by Scott Ross
Something to remember about omniorb is that will use different
transports if they are available. By default it will use Unix
domain sockets on the localhost and TCP(or SSL) when going remote.

I mentioned elsewhere that I'm specifying the transport using
-ORBendPoint options, TCP for both localhost and remote.

Post by Scott Ross
might affect the results -- calls/connection, connection/server,
threads/connection, POA policies, etc. But they (for me) have never
made 'order of magnitude' differences.

I don't know that it's necessarily bad, as PING is seeing quite a
delay as well. But what I can't explain is what gSOAP doesn't see
a similar latency hit. I have to put together a coherent story
here pretty soon.

I'm doing everything single-threaded, as there's only one client
and one server, so I have to worry about context switch.


Brian



__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Duncan Grisby]

Post by Default User
I don't know that it's necessarily bad, as PING is seeing quite a
delay as well. But what I can't explain is what gSOAP doesn't see
a similar latency hit. I have to put together a coherent story
here pretty soon.

Does gSOAP perhaps stream multiple requests, rather than waiting for
each one to be replied to? If so, it's bound by the bandwidth of the
network rather than the latency. That's more like the situation with
CORBA oneway calls.

If you're going to try doing oneways, you should set the
maxServerThreadPerConnection parameter to 1 to avoid omniORB trying to
handle multiple requests concurrently and ending up doing lots of thread
switching.

Cheers,

Duncan.

[Default User]

Sorry, I sometimes forget to set the replies to go to the mailing
list rather than the sender.

Post by Duncan Grisby
Does gSOAP perhaps stream multiple requests, rather than waiting for
each one to be replied to? If so, it's bound by the bandwidth of the
network rather than the latency. That's more like the situation with
CORBA oneway calls.

That's quite possible. I'll look through the settings.

Post by Duncan Grisby
If you're going to try doing oneways, you should set the
maxServerThreadPerConnection parameter to 1 to avoid omniORB trying
to handle multiple requests concurrently and ending up doing lots of
thread switching.

Ok, thanks for the information. I'll look at that situation. I want
to get a reasonable comparison setup.




__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Default User]

Post by Duncan Grisby
Can you post some more details about what your test is actually
doing?
How much data are you sending in each call? What kind of data is it?
Are you just doing simple calls, or something more exotic with
callbacks or something?

I use pretty standard setup from the examples, except that I use
explicit endpoints always, for both client and server.

Here's one example of one the tests.

IDL:

typedef sequence<double> DoubleArray;
interface FTSService
{
short DoubleArrayIn(in DoubleArray array);
};


Server implementation:

short FTSService_i::DoubleArrayIn(const DoubleArray& array)
{
return 0;
}


Client:

//"size" and "iterations" are set by prompts to the user
data = new CORBA::Double[size];

DoubleArray array(size, size, data);
for (size_t j = 0; j < size; j++)
array[j] = j+1;

InitTimer(); // Our timing method, also tried gettimeofday()

for (long i = 0; i < iterations; i++)
{
res = FTSServiceRef->DoubleArrayIn(array);
}

double t = GetElapsedTime(); //see InitTimer()

printf("\nMessage: %ld Time: %.3f Average: %.8f \n\n",
iterations, t, t/iterations);

delete[] data;

Post by Duncan Grisby
What kind of absolute times do you get for your calls?

I have been running 10000 iterations per data set.

For an array of 10 doubles, I get about .23 seconds total, or an
average of .000023 seconds each.

For 1000 doubles, it's .36/.000036.



Brian


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com