[omniORB] Marshall exception -- message size limit exceeded

[Henrique Andrade]

Hi:

Let me describe the problem I am facing as well as an application scenario.

Problem: not being able to detect that the message size limit has been
reached via server/client-side exceptions. Detection cannot be done at the
server nor can it be done at the client. In other words, failure occurs and
cannot be detected (tested in omniORB 4.0.7 and 4.1).

Scenario: an RPC call is made to the server and the server's response has a
varying size, which can potentially go over the limit giopMaxMsgSize

In this scenario, on the server side, an internal MARSHAL exception
eventually happens once the server detects the message is beyond the
giopMaxMsgSize
limit and is handled internally (ORB tracing confirms that) by one of the
ORB threads, but it does not reach the application. That means, the app
fails, but there is no way for it to know a failure occurred.

On the client, side a COMM_FAILURE exception will happen. However, the minor
code does not indicate MARSHAL_MessageSizeExceedLimit (it indicates
COMM_FAILURE_WaitingForReply instead). So the client does not know why the
failure occurred either -- that is that the response is larger than what can
be handled.

Note that this behavior seems to be asymmetrical. That is, if the client
makes an RPC which sends a parameter whose size is greater than giopMaxMsgSize,
the client gets a MARSHAL exception with the correct minor code and can
react accordingly as expected.

On the original problem I described, is there anything that can be done to
allow the detection of that failure as well as the actual reason (as in the
minor code) so at least the problem can be logged? If not, would it be
possible for me to file a feature request?

Thanks,

Henrique
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.omniorb-support.com/pipermail/omniorb-list/attachments/20070129/2aa0d8be/attachment.htm

[Luke Deller]

Post by Henrique Andrade
On the client, side a COMM_FAILURE exception will happen. However, the
minor code does not indicate MARSHAL_MessageSizeExceedLimit (it

indicates

Post by Henrique Andrade
COMM_FAILURE_WaitingForReply instead). So the client does not know why

the

Post by Henrique Andrade
failure occurred either -- that is that the response is larger than

what

Post by Henrique Andrade
can be handled.

I see the following comment in src/lib/omniORB/orbcore/giopImpl12.cc, in
the function giopImpl12::sendSystemException

// This system exception is raised during the marshalling of the
reply.
// We cannot marshal the exception. Can only indicate that something
// fatal about this request.

I suppose it makes sense that the server can't change its mind half way
through sending a response. It should be possible to calculate the
size of a response *before* starting to send it, but that might be
expensive.

Here's a question of my own:
Why does omniORB limit the size of outgoing messages anyway? I can
understand limiting the size of incoming messages, but for outgoing
messages the application code is in a position to decide for itself
whether to send a certain client call or server reply, right?

Luke.
**********************************************************************************************
Important Note
This email (including any attachments) contains information which is confidential and may be subject to legal privilege. If you are not the intended recipient you must not use, distribute or copy this email. If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this email are not necessarily the views of XPlan Technology.

It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system.
Xplan Technology does not warrant that the information is free of a virus or any other defect or error.
**********************************************************************************************

[Duncan Grisby]

Post by Henrique Andrade

Post by Henrique Andrade
On the client, side a COMM_FAILURE exception will happen. However, the
minor code does not indicate MARSHAL_MessageSizeExceedLimit (it

indicates

Post by Henrique Andrade
COMM_FAILURE_WaitingForReply instead). So the client does not know why

the

Post by Henrique Andrade
failure occurred either -- that is that the response is larger than

what

Post by Henrique Andrade
can be handled.

I see the following comment in src/lib/omniORB/orbcore/giopImpl12.cc, in
the function giopImpl12::sendSystemException
// This system exception is raised during the marshalling of the
reply.
// We cannot marshal the exception. Can only indicate that something
// fatal about this request.
I suppose it makes sense that the server can't change its mind half way
through sending a response. It should be possible to calculate the
size of a response *before* starting to send it, but that might be
expensive.

Yes -- that's what happens. omniORB does not normally pre-calculate
message sizes, because to do so involves basically doing all the
marshalling work twice. Because of that, it doesn't know whether the
message is going to be too big until it has started sending the message.

Post by Henrique Andrade
Why does omniORB limit the size of outgoing messages anyway? I can
understand limiting the size of incoming messages, but for outgoing
messages the application code is in a position to decide for itself
whether to send a certain client call or server reply, right?

Two reasons really. First, if clients are configured to limit the
incoming message size, it makes sense for servers to similarly limit the
messages to avoid sending more data than the client will accept.
Secondly, and probably more importantly, it guards against unpleasant
behaviour in the face of memory corruption problems. Imagine you're
sending a sequence and memory corruption sets the sequence length to
2^32 items. It's probably a bad thing for omniORB to attempt to send all
that data.

It would probably make sense to have separate sending and receiving
limits. Anyone want to make a patch for it?

Cheers,

Duncan.

[Philippe Selo]

[re-send to the list]

Duncan and Luke,

Thanks for you answers.

The issue that Henrique was describing is not so much the limit to the
message size as the handling on the server side.
In our system, we have a wrapper on the client side that catches the
COMM_FAILURE exception and retries based on client policies (we
experience a lot of comm failures when we run with a large number of
machines).
When the server's answer to a request is larger than the max size of
the buffer, it sends a very generic COMM_FAILURE and our client
retries again, which is useless. It would help a lot if the server was
sending a more specific exception that the client could distinguish
from a comm failure.

Regards,
Philippe.

Post by Duncan Grisby

Post by Henrique Andrade

Post by Henrique Andrade
On the client, side a COMM_FAILURE exception will happen. However, the
minor code does not indicate MARSHAL_MessageSizeExceedLimit (it

indicates

Post by Henrique Andrade
COMM_FAILURE_WaitingForReply instead). So the client does not know why

the

Post by Henrique Andrade
failure occurred either -- that is that the response is larger than

what

Post by Henrique Andrade
can be handled.

I see the following comment in src/lib/omniORB/orbcore/giopImpl12.cc, in
the function giopImpl12::sendSystemException
// This system exception is raised during the marshalling of the
reply.
// We cannot marshal the exception. Can only indicate that something
// fatal about this request.
I suppose it makes sense that the server can't change its mind half way
through sending a response. It should be possible to calculate the
size of a response *before* starting to send it, but that might be
expensive.

Yes -- that's what happens. omniORB does not normally pre-calculate
message sizes, because to do so involves basically doing all the
marshalling work twice. Because of that, it doesn't know whether the
message is going to be too big until it has started sending the message.

Post by Henrique Andrade
Why does omniORB limit the size of outgoing messages anyway? I can
understand limiting the size of incoming messages, but for outgoing
messages the application code is in a position to decide for itself
whether to send a certain client call or server reply, right?

Two reasons really. First, if clients are configured to limit the
incoming message size, it makes sense for servers to similarly limit the
messages to avoid sending more data than the client will accept.
Secondly, and probably more importantly, it guards against unpleasant
behaviour in the face of memory corruption problems. Imagine you're
sending a sequence and memory corruption sets the sequence length to
2^32 items. It's probably a bad thing for omniORB to attempt to send all
that data.
It would probably make sense to have separate sending and receiving
limits. Anyone want to make a patch for it?
Cheers,
Duncan.
--
-- Duncan Grisby --
-- http://www.grisby.org --
_______________________________________________
omniORB-list mailing list
http://www.omniorb-support.com/mailman/listinfo/omniorb-list

[Luke Deller]

Post by Philippe Selo
The issue that Henrique was describing is not so much the limit to
the message size as the handling on the server side.

Duncan's suggestion to have separate configurable limits for sending and
receiving would actually solve your problem.

It's not obvious how to improve the handling on the server site for the
reasons that we raised. However, if you could set the sending limit to
be higher than the receiving limit, then the receiving limit would be
exceeded first. The receiver is able to raise a useful error message
for you (ie MARSHAL_MessageSizeExceedLimit instead of COMM_FAILURE).

Regards,
Luke.
**********************************************************************************************
Important Note
This email (including any attachments) contains information which is confidential and may be subject to legal privilege. If you are not the intended recipient you must not use, distribute or copy this email. If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this email are not necessarily the views of XPlan Technology.

It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system.
Xplan Technology does not warrant that the information is free of a virus or any other defect or error.
**********************************************************************************************

[Philippe Selo]

Luke,

Maybe I'm missing something. The scenario that is causing us trouble
is when the client send a tiny request, like "status" and the server
is trying to send a very large response. the server get an exception,
regardless of the setting of the client.

An IDL like that:

module test {

interface bla {

void status(out string result) ;
} ;

} ;

would exhibit the problem. Of course we can raise the max size limit
on the client and server but if the response is arbitrary large, there
is always a chance of exceeding the limit, so I think it can be
helpful for the client to "know" that it will never get the answer,
even if it retry.

regards,
Philippe

Post by Luke Deller

Post by Philippe Selo
The issue that Henrique was describing is not so much the limit to
the message size as the handling on the server side.

Duncan's suggestion to have separate configurable limits for sending and
receiving would actually solve your problem.
It's not obvious how to improve the handling on the server site for the
reasons that we raised. However, if you could set the sending limit to
be higher than the receiving limit, then the receiving limit would be
exceeded first. The receiver is able to raise a useful error message
for you (ie MARSHAL_MessageSizeExceedLimit instead of COMM_FAILURE).
Regards,
Luke.
**********************************************************************************************
Important Note
This email (including any attachments) contains information which is confidential and may be subject to legal privilege. If you are not the intended recipient you must not use, distribute or copy this email. If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this email are not necessarily the views of XPlan Technology.
It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system.
Xplan Technology does not warrant that the information is free of a virus or any other defect or error.
**********************************************************************************************
_______________________________________________
omniORB-list mailing list
http://www.omniorb-support.com/mailman/listinfo/omniorb-list

[Luke Deller]

Post by Philippe Selo
Maybe I'm missing something.

...

Post by Philippe Selo
Of course we can raise the max size limit on the client and
server but if the response is arbitrary large, there is
always a chance of exceeding the limit, so I think it can
be helpful for the client to "know" that it will never get
the answer, even if it retry.

I understand that you want the client to know why the request failed
when a message size limit is exceeded. I'm saying that this can be
achieved if the message size limit is exceeded in the right place: in
the client as it receives the response, rather than in the server as it
sends the response.

Let me try to explain what I mean with two examples.

Example 1:
- giopMaxMsgSize is set to 4MB in server and client processes
- your function tries to return a 50Mb result

In this example, the server process will start sending the response to
the client. When the server realises that the response is >4Mb, it will
abort sending the response, triggering a COMM_FAILURE. Unfortunately as
we have discussed there is no way for the server to raise a more useful
exception at this stage because it has already started sending a
response. This outcome is not what you want.

Example 2:
- giopMaxRecvMsgSize is set to 4Mb in server and client processes
- giopMaxSendMsgSize is set to 32Mb in server and client processes
- your function tries to return a 50Mb result

Again, the server process will start sending the response to the client.
The server will happily keep sending past 4Mb of data, but when the
client realises that the response it is receiving is >4Mb it will abort
receiving the response, and raise a MARSHAL exception with a descriptive
minor code. This is the outcome that you want.

The second example only works if somebody implements giopMaxRecvMsgSize
and giopMaxRecvMsgSize settings in omniORB.

Regards,
Luke.
**********************************************************************************************
Important Note
This email (including any attachments) contains information which is confidential and may be subject to legal privilege. If you are not the intended recipient you must not use, distribute or copy this email. If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this email are not necessarily the views of XPlan Technology.

It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system.
Xplan Technology does not warrant that the information is free of a virus or any other defect or error.
**********************************************************************************************

[Philippe Selo]

Luke,

thanks for the clarification, I missed Duncan's point about the two
buffer limits.

p.

Post by Luke Deller

Post by Philippe Selo
Maybe I'm missing something.

...

Post by Philippe Selo
Of course we can raise the max size limit on the client and
server but if the response is arbitrary large, there is
always a chance of exceeding the limit, so I think it can
be helpful for the client to "know" that it will never get
the answer, even if it retry.

I understand that you want the client to know why the request failed
when a message size limit is exceeded. I'm saying that this can be
achieved if the message size limit is exceeded in the right place: in
the client as it receives the response, rather than in the server as it
sends the response.
Let me try to explain what I mean with two examples.
- giopMaxMsgSize is set to 4MB in server and client processes
- your function tries to return a 50Mb result
In this example, the server process will start sending the response to
the client. When the server realises that the response is >4Mb, it will
abort sending the response, triggering a COMM_FAILURE. Unfortunately as
we have discussed there is no way for the server to raise a more useful
exception at this stage because it has already started sending a
response. This outcome is not what you want.
- giopMaxRecvMsgSize is set to 4Mb in server and client processes
- giopMaxSendMsgSize is set to 32Mb in server and client processes
- your function tries to return a 50Mb result
Again, the server process will start sending the response to the client.
The server will happily keep sending past 4Mb of data, but when the
client realises that the response it is receiving is >4Mb it will abort
receiving the response, and raise a MARSHAL exception with a descriptive
minor code. This is the outcome that you want.
The second example only works if somebody implements giopMaxRecvMsgSize
and giopMaxRecvMsgSize settings in omniORB.
Regards,
Luke.
**********************************************************************************************
Important Note
This email (including any attachments) contains information which is confidential and may be subject to legal privilege. If you are not the intended recipient you must not use, distribute or copy this email. If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this email are not necessarily the views of XPlan Technology.
It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system.
Xplan Technology does not warrant that the information is free of a virus or any other defect or error.
**********************************************************************************************