[omniORB] returning a long sequence - is there a limit ?

[Ilya Zvyagin]

Hello, and first of all thank you for a good product.

I have a problem passing a long sequence of objects
to the client while shorter sequence works.

I use omniOrb 4.1.1 on Ubuntu Linux 2.6.22-14-generic x86.

I wrote a test application to test the idea we are going
to use in our future application a lot.
This is a CORBA server application which fetches some
dataset from database and builds a sequence of structures
(CORBA classes) representing each row of the dataset.
The sequence is returned then to the client application
as result of this method. This is an unbounded sequence.

This is the IDL:

#pragma prefix "util.rlisystems.ru"

module costress {

typedef string<21> datetime; // date in format 20011231 01:02:13:000
valuetype Itemser
{
public long long object_id;
public long long class_id;
public string name;
public string shortname;
public long long item_id;
public long long vat_id;
public boolean is_default;
public string barcode;
public long long lifetime_id;
public datetime created;
public long long usr_id;
public string login_name;
public string unit_name;

factory MakeItemser (in long long object_id,
in long long class_id,
in string name,
in string shortname,
in long long item_id,
in long long vat_id,
in boolean is_default,
in string barcode,
in long long lifetime_id,
in datetime created,
in long long usr_id,
in string login_name,
in string unit_name );
};

typedef sequence< Itemser > ItemserSet;

interface ICoStress {
ItemserSet allGet();
};

};

-----------------------------------------------------------------------------
The application seems to work well and returns needed
data to the client. But there is some problem as there
is a limit on the number of rows returned.
If I return up to 9019 rows, all goes fine, but if I try
to return 9020 rows or more, a GIOP protocol error exception
is raised and the call fails.

To find out this, I added a fetch limit parameter to the server
application and, after experiments, found this. Actually
the dataset contains something about 30,000 rows and my
final goal is to try to send even such bug results.

-r parameter of costress server app defines maximum number of rows
to fetch and return. -r 0 means "no limit".

This is the transcript:
*******************************************************************************

***@host:~$ dist/Debug/GNU-Linux-x86/costress -S srv -U usr -D database -P
****** -r 0 &
[1] 6122

***@host:~$ ../costress_client/dist/Debug/GNU-Linux-x86/costress_client `cat
costress.ior`

omniORB: To endpoint: giop:tcp:[::ffff:192.168.27.49]:43377. Send GIOP 1.2
MessageError because a protocol error has been detected. Connection is closed.
omniORB: From endpoint: giop:tcp:192.168.27.49:37010. Detected GIOP 1.2 protocol
error in input message. giopImpl12.cc:289. Connection is closed.
terminate called after throwing an instance of 'CORBA::COMM_FAILURE'
calling allGet() ... Aborted (core dumped)

***@host:~$ fg
dist/Debug/GNU-Linux-x86/costress -S roo -U sa -D POLYGON -P control -r 0
(C-c)

***@host:~$ dist/Debug/GNU-Linux-x86/costress -S srv -U usr -D database -P
****** -r 9019 &
[1] 6132

***@host:~$ ../costress_client/dist/Debug/GNU-Linux-x86/costress_client `cat
costress.ior`

calling allGet() ... allGet() call took 8 sec, size returned is 9019

*******************************************************************************

So the questions are:
0) Is there a limit on length of sequences returned either in CORBA spec.
or omniORB implementation ?
(I think there should not be any).

1) How can I further investigate and solve this problem ?

Thank you.

[evgeni.rojkov at durr.com ()]

Hier is a part from omniOrb.cfg.
Is giopMaxMsgSize what are you looking for ?
Regards, Evgeni

# giopMaxMsgSize
#
# This value defines the ORB-wide limit on the size of GIOP message
# (excluding the header). If this limit is exceeded, the ORB will
# refuse to send or receive the message and raise a MARSHAL exception.
#
# Valid values = (n >= 8192)
#
giopMaxMsgSize = 2097152 # 2 MBytes.



-----Urspr?ngliche Nachricht-----
Von: omniorb-list-***@omniorb-support.com
[mailto:omniorb-list-***@omniorb-support.com] Im Auftrag von Ilya Zvyagin
Gesendet: Donnerstag, 24. Januar 2008 15:44
An: omniorb-***@omniorb-support.com
Betreff: [omniORB] returning a long sequence - is there a limit ?


Hello, and first of all thank you for a good product.

I have a problem passing a long sequence of objects
to the client while shorter sequence works.

I use omniOrb 4.1.1 on Ubuntu Linux 2.6.22-14-generic x86.

I wrote a test application to test the idea we are going
to use in our future application a lot.
This is a CORBA server application which fetches some
dataset from database and builds a sequence of structures (CORBA classes)
representing each row of the dataset. The sequence is returned then to the
client application as result of this method. This is an unbounded sequence.

This is the IDL:

#pragma prefix "util.rlisystems.ru"

module costress {

typedef string<21> datetime; // date in format 20011231 01:02:13:000
valuetype Itemser
{
public long long object_id;
public long long class_id;
public string name;
public string shortname;
public long long item_id;
public long long vat_id;
public boolean is_default;
public string barcode;
public long long lifetime_id;
public datetime created;
public long long usr_id;
public string login_name;
public string unit_name;

factory MakeItemser (in long long object_id,
in long long class_id,
in string name,
in string shortname,
in long long item_id,
in long long vat_id,
in boolean is_default,
in string barcode,
in long long lifetime_id,
in datetime created,
in long long usr_id,
in string login_name,
in string unit_name );
};

typedef sequence< Itemser > ItemserSet;

interface ICoStress {
ItemserSet allGet();
};

};

-----------------------------------------------------------------------------
The application seems to work well and returns needed
data to the client. But there is some problem as there
is a limit on the number of rows returned.
If I return up to 9019 rows, all goes fine, but if I try
to return 9020 rows or more, a GIOP protocol error exception
is raised and the call fails.

To find out this, I added a fetch limit parameter to the server application and,
after experiments, found this. Actually the dataset contains something about
30,000 rows and my final goal is to try to send even such bug results.

-r parameter of costress server app defines maximum number of rows to fetch and
return. -r 0 means "no limit".

This is the transcript:
*******************************************************************************

***@host:~$ dist/Debug/GNU-Linux-x86/costress -S srv -U usr -D database -P
****** -r 0 &
[1] 6122

***@host:~$ ../costress_client/dist/Debug/GNU-Linux-x86/costress_client `cat
costress.ior`

omniORB: To endpoint: giop:tcp:[::ffff:192.168.27.49]:43377. Send GIOP 1.2
MessageError because a protocol error has been detected. Connection is closed.
omniORB: From endpoint: giop:tcp:192.168.27.49:37010. Detected GIOP 1.2 protocol
error in input message. giopImpl12.cc:289. Connection is closed. terminate
called after throwing an instance of 'CORBA::COMM_FAILURE' calling allGet() ...
Aborted (core dumped)

***@host:~$ fg
dist/Debug/GNU-Linux-x86/costress -S roo -U sa -D POLYGON -P control -r 0
(C-c)

***@host:~$ dist/Debug/GNU-Linux-x86/costress -S srv -U usr -D database -P
****** -r 9019 &
[1] 6132

***@host:~$ ../costress_client/dist/Debug/GNU-Linux-x86/costress_client `cat
costress.ior`

calling allGet() ... allGet() call took 8 sec, size returned is 9019

*******************************************************************************

So the questions are:
0) Is there a limit on length of sequences returned either in CORBA spec. or
omniORB implementation ? (I think there should not be any).

1) How can I further investigate and solve this problem ?

Thank you.

[Ilya Zvyagin]

Post by evgeni.rojkov at durr.com ()
Hier is a part from omniOrb.cfg.
Is giopMaxMsgSize what are you looking for ?
Regards, Evgeni
# giopMaxMsgSize
#
# This value defines the ORB-wide limit on the size of GIOP message
# (excluding the header). If this limit is exceeded, the ORB will
# refuse to send or receive the message and raise a MARSHAL exception.
#
# Valid values = (n >= 8192)
#
giopMaxMsgSize = 2097152 # 2 MBytes.

Yes, may be very well. Thank you, I 'll try this.

[Ilya Zvyagin]

Post by evgeni.rojkov at durr.com ()
Hier is a part from omniOrb.cfg.
Is giopMaxMsgSize what are you looking for ?
Regards, Evgeni
# giopMaxMsgSize

Thank you, this helped. I increased this parameter something like
2.5-3 times and my hapless result set passed to the client.
Thanks to all took part in the thread.

****
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.omniorb-support.com/pipermail/omniorb-list/attachments/20080125/943b48dd/attachment.htm

[Thomas Lockhart]

Post by Ilya Zvyagin
1) How can I further investigate and solve this problem ?

Since you want to support an unlimited amount of data, the more robust
solution would be to return the result with an iterator. That way both
client and server have some influence over the maximum transfer size
each can handle. The iterator interface can also allow the entire result
to be returned in the first call, if the result is smaller than the
maximum size.

I'm not recalling if there are examples of iterators in standard CORBA
interfaces or in the omniORB example code. If you can not find one then
ask on the list.

hth

- Tom

[Ilya Zvyagin]

Post by Thomas Lockhart

Post by Ilya Zvyagin
1) How can I further investigate and solve this problem ?

Since you want to support an unlimited amount of data, the more robust
solution would be to return the result with an iterator. That way both
client and server have some influence over the maximum transfer size
each can handle. The iterator interface can also allow the entire
result to be returned in the first call, if the result is smaller than
the maximum size.

Thank you for idea, but I think returning one big sequence would avoid
many round trips
from server to client and back and this way could cause performance
degradation.
My idea was not to have enormous data sets returned, like millions of
rows, this is just
ridiculous, but only some reasonably big results, like 100-1000. And the
idea of jumping
to 30k rows dataset was only to try testing overhead.

[Thomas Lockhart]

Post by Ilya Zvyagin
Thank you for idea, but I think returning one big sequence would
avoid many round trips from server to client and back and this way
could cause performance degradation. My idea was not to have enormous
data sets returned, like millions of rows, this is just ridiculous,
but only some reasonably big results, like 100-1000. And the idea of
jumping to 30k rows dataset was only to try testing overhead.

If the iterator interface is designed to return a (partial) result in
the first call, then you will not have multiple calls to handle
"reasonably big results". However, if you are returning results from,
say, a DB, you will have no absolute control over the largest return
set, and could run into fundamental system limitations which cause
transfer failures. So designing with an iterator may be considered a
more robust (and in the typical case, a no more expensive) solution.

hth

- Tom

[BaileyK at schneider.com ()]

CosNaming::NamingContext has a list() method that returns an iterator.


---
Kendall Bailey
Engineering & Research
Schneider National, Inc.




Thomas Lockhart
<***@fourpal
ms.org> To
Sent by: ***@gmail.com
omniorb-list-boun cc
***@omniorb-suppo omniorb-***@omniorb-support.com
rt.com Subject
Re: [omniORB] returning a long
sequence - is there a limit ?
01/24/2008 09:43
AM

Post by Ilya Zvyagin
1) How can I further investigate and solve this problem ?

Since you want to support an unlimited amount of data, the more robust
solution would be to return the result with an iterator. That way both
client and server have some influence over the maximum transfer size
each can handle. The iterator interface can also allow the entire result
to be returned in the first call, if the result is smaller than the
maximum size.

I'm not recalling if there are examples of iterators in standard CORBA
interfaces or in the omniORB example code. If you can not find one then
ask on the list.

hth

- Tom