[omniORB] OmniOrb and CP1252 (Windows Latin 1) vs. ISO-8859-1

[Likun Yuan]

Hi All,

Following Duncan's instruction, I have built the CP1252 support into
OmniORB 4.1.1. (I'd like to volunteer myself to build the other codesets
tables after I made sure this one's working properly. :-))

1. I added the generated cs-cp1252.cc into the omniCodeSets411_rt.lib,
and linked my application with this lib. The omniORB parameters dumped
at the starting time are:
omniORB: Version: 4.1.1
omniORB: Distribution date: Sun Oct 7 16:41:47 BST 2007 dgrisby
omniORB: My addresses are:
omniORB: 172.16.2.143
omniORB: 127.0.0.1
omniORB: The endPointPublishAllIFs parameter is deprecated.
omniORB: Use an endPointPublish specification instead.
omniORB: Maximum supported GIOP version is 1.2
omniORB: Native char code sets: GBK SNI-EDF-4 IBM-500 IBM-037
windows-1252 windows-1251 ISO-8859-10 ISO-8859-9 ISO-8859-8 ISO-8859-7
ISO-8859-6 ISO-8859-5 ISO-8859-4 ISO-8859-3 ISO-8859-2 UTF-8 ISO-8859-1.
omniORB: Transmission char code sets: GBK(1.2) GBK(1.1) SNI-EDF-4(1.2)
SNI-EDF-4(1.1) IBM-500(1.2) IBM-500(1.1) IBM-037(1.2) IBM-037(1.1)
windows-1252(1.2) windows-1251(1.2) ISO-8859-10(1.2) ISO-8859-9(1.2)
ISO-8859-8(1.2) ISO-8859-7(1.2) ISO-8859-6(1.2) ISO-8859-5(1.2)
ISO-8859-4(1.2) ISO-88
2) ISO-8859-2(1.2) UTF-8(1.2) UTF-8(1.1) ISO-8859-1(1.2) ISO-8859-1(1.1)
ISO-8859-1(1.0).
omniORB: Native wide char code sets: UTF-16.
omniORB: Transmission wide char code sets: UTF-16(1.2).
omniORB: Information: the omniDynamic library is not linked.
omniORB: Current configuration is as follows:
omniORB: DefaultInitRef (file) =
omniORB: DefaultInitRef (args) =
omniORB: InitRef = NameService=corbaname::172.16.2.143:12345
omniORB: abortOnInternalError = 0
omniORB: abortOnNativeException = 0
omniORB: acceptBiDirectionalGIOP = 0
omniORB: acceptMisalignedTcIndirections = 0
omniORB: bootstrapAgentHostname =
omniORB: bootstrapAgentPort = 900
omniORB: clientCallTimeOutPeriod = 0
omniORB: clientConnectTimeOutPeriod = 0
omniORB: clientTransportRule = * unix,ssl,tcp
omniORB: configFile = [none]
omniORB: connectionWatchImmediate = 0
omniORB: connectionWatchPeriod = 50000
omniORB: copyValuesInLocalCalls = 1
omniORB: diiThrowsSysExceptions = 0
omniORB: dumpConfiguration = 1
omniORB: endPoint = giop:tcp::
omniORB: endPointPublish = addr,all(addr)
omniORB: endPointPublishAllIFs = 1
omniORB: giopMaxMsgSize = 2097152
omniORB: giopTargetAddressMode = KeyAddr
omniORB: id = omniORB4
omniORB: idleThreadTimeout = 10
omniORB: inConScanPeriod = 180
omniORB: lcdMode = 0
omniORB: maxGIOPConnectionPerServer = 5
omniORB: maxGIOPVersion = 1.2
omniORB: maxInterleavedCallsPerConnection = 5
omniORB: maxServerThreadPerConnection = 100
omniORB: maxServerThreadPoolSize = 100
omniORB: maxSocketRecv = 131072
omniORB: maxSocketSend = 131072
omniORB: nativeCharCodeSet = windows-1252
omniORB: nativeWCharCodeSet = UTF-16
omniORB: objectTableSize = 0
omniORB: offerBiDirectionalGIOP = 0
omniORB: oneCallPerConnection = 1
omniORB: outConScanPeriod = 120
omniORB: poaHoldRequestTimeout = 0
omniORB: poaUniquePersistentSystemIds = 1
omniORB: principal = [Null]
omniORB: resetTimeOutOnRetries = 0
omniORB: scanGranularity = 5
omniORB: serverCallTimeOutPeriod = 0
omniORB: serverTransportRule = * unix,ssl,tcp
omniORB: socketSendBuffer = 16384
omniORB: strictIIOP = 1
omniORB: supportBootstrapAgent = 0
omniORB: supportCurrent = 1
omniORB: supportPerThreadTimeOut = 0
omniORB: tcAliasExpand = 0
omniORB: threadPerConnectionLowerLimit = 9000
omniORB: threadPerConnectionPolicy = 1
omniORB: threadPerConnectionUpperLimit = 10000
omniORB: threadPoolWatchConnection = 1
omniORB: traceExceptions = 1
omniORB: traceFile = [stderr]
omniORB: traceInvocationReturns = 0
omniORB: traceInvocations = 0
omniORB: traceLevel = 25
omniORB: traceThreadId = 0
omniORB: traceTime = 0
omniORB: unixTransportDirectory = /tmp/omni-%u
omniORB: unixTransportPermission = 777
omniORB: useTypeCodeIndirections = 1
omniORB: verifyObjectExistsAndType = 1

2. I have another Java ORB application, with the resolved IOR:
Type ID: "IDL:ConnectionBroker/CBFunctions:1.0"
Profiles:
1. IIOP 1.2 NPS00004 12356 ".......
....................RootPOA.....CB_poa..............."
TAG_CODE_SETS char native code set: UTF-8
char conversion code set: UTF-8
wchar native code set: UTF-16
wchar conversion code set: 0x05010001

When the JAVA ORB sending the data to the OmniORB application, I was
expecting the OmniORB application set the TCS to windows-1252 based on
the NativeCharCodeSet configuration. But I actually got [UTF-8], the
OmniORB log message as below.

10/14/2008 16:06:07 4332:4108 [DEBUG] omniORBLog(): OmniORB log message:
omniORB: Receive codeset service context and set TCS to (UTF-8,UTF-16)

Is there anything I can do to force the OmniORB to set the TCS to
(windows-1252,UTF-16)? I can't do it on the Java side, the JAVA ORB
doesn't support the windows-1252 codeset at all.

Thank you very much in advance!

...

If you want to make a CP1252 codeset for omniORB, you can automatically
generate the tables using bin/scripts/make8bitcs.py giving it input

from

http://www.unicode.org/Public/MAPPINGS/

ftp://ftp.opengroup.org/pub/code_set_registry/code_set_registry1.2g.txt

Any volunteers to make a patch containing all the tables for the
additional ISO 8859 and Windows codesets in it?
Cheers,
Duncan.
--
-- Duncan Grisby --
-- duncan at grisby.org

<http://www.omniorb-support.com/mailman/listinfo/omniorb-list> --

-- http://www.grisby.org --

_____________________________
Likun Yuan
Server Team Lead
North Plains Systems Corp.
P: 416 345 1900 x 507
***@northplains.com

www.northplains.com
"Bringing FOCUS to Digital Asset Management"
Confidentiality Notice:
The information contained herein is confidential and proprietary to
North Plains Systems Corp. ("North Plains") and is intended for review
by authorized persons only. Except as may otherwise be agreed to in
writing by North Plains, any disclosure, circulation, release or use of
the information contained herein is strictly prohibited



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.omniorb-support.com/pipermail/omniorb-list/attachments/20081014/ca494057/attachment-0001.htm

[Duncan Grisby]

Following Duncan?s instruction, I have built the CP1252 support into OmniORB
4.1.1. (I?d like to volunteer myself to build the other codesets tables after
I made sure this one?s working properly. J)

I beat you to it :-) omniORB 4.1.3 has all the Windows code sets, plus
all the ISO-8859 ones that were missing.

[...]

omniORB: nativeCharCodeSet = windows-1252

[...]

Type ID: "IDL:ConnectionBroker/CBFunctions:1.0"
1. IIOP 1.2 NPS00004 12356 ".......
....................RootPOA.....CB_poa..............."
TAG_CODE_SETS char native code set: UTF-8
char conversion code set: UTF-8
wchar native code set: UTF-16
wchar conversion code set: 0x05010001

This tells us that the Java ORB only knows about UTF-8 for char data.

When the JAVA ORB sending the data to the OmniORB application, I was
expecting the OmniORB application set the TCS to windows-1252 based on
the NativeCharCodeSet configuration. But I actually got [UTF-8], the
OmniORB log message as below.

TCS is the Transmission Code Set. It has to be something both ORBs
understand. Since the Java ORB doesn't understand windows-1252, it
chooses UTF-8, which both ends understand.

omniORB: Receive codeset service context and set TCS to (UTF-8,UTF-16)
Is there anything I can do to force the OmniORB to set the TCS to
(windows-1252,UTF-16)? I can?t do it on the Java side, the JAVA ORB doesn?t
support the windows-1252 codeset at all.

You can't use windows-1252 as the Transmission Code Set, but you _are_
using it as the Native Code Set. omniORB will translate string data to
and from UTF-8 for you. That's the whole point of the code set
negotiation and translation scheme.

Cheers,

Duncan.

[Likun Yuan]

On Tuesday 14 October, "Duncan Grisby" wrote:
...

Post by Duncan Grisby
You can't use windows-1252 as the Transmission Code Set, but you _are_
using it as the Native Code Set. omniORB will translate string data to
and from UTF-8 for you. That's the whole point of the code set
negotiation and translation scheme.

You're absolutely right; omniORB did translate the string data to CP1252 for me. :-) Now I can send the Euro "?" between the ORBs, supper...

...

Post by Duncan Grisby
I beat you to it :-) omniORB 4.1.3 has all the Windows code sets, plus
all the ISO-8859 ones that were missing.

COOL... an extra question, what's new features and major fix included in the 4.1.3 build? I remembered that a couple of weeks ago, when I was planning to upgrade to omnoORB 4.1.x, I did some research, somebody from a website said that the 4.1.2 version had file handle leak some how. So I have downloaded the 4.1.1 version. Did you notice such a bug in 4.1.2 version? Was it fixed in the new release? I would like to keep up with the latest omniORB build; however I have to consider the stabilities of the products first.

Thanks a bunch!!!

_____________________________
Likun Yuan
Server Team Lead
North Plains Systems Corp.
P: 416 345 1900 x 507
***@northplains.com

www.northplains.com
"Bringing FOCUS to Digital Asset Management"
Confidentiality Notice:
The information contained herein is confidential and proprietary to North Plains Systems Corp. ("North Plains") and is intended for review by authorized persons only. Except as may otherwise be agreed to in writing by North Plains, any disclosure, circulation, release or use of the information contained herein is strictly prohibited

[Duncan Grisby]

Post by Likun Yuan
COOL... an extra question, what's new features and major fix included
in the 4.1.3 build?

The release notes are here:

http://sourceforge.net/project/shownotes.php?release_id=629708&group_id=51138

and the list of bugs fixed is here:

http://omniorb.sourceforge.net/bugs/bugfixes-412.html
http://omniorb.sourceforge.net/bugs/bugfixes-411.html

Post by Likun Yuan
I remembered that a couple of weeks ago, when I was planning to
upgrade to omnoORB 4.1.x, I did some research, somebody from a website
said that the 4.1.2 version had file handle leak some how. So I have
downloaded the 4.1.1 version. Did you notice such a bug in 4.1.2
version? Was it fixed in the new release? I would like to keep up
with the latest omniORB build; however I have to consider the
stabilities of the products first.

Nobody has reported a file handle leak in 4.1.2. A quick web search for
"omniorb 4.1.0 file handle leak" didn't find anything. There certainly
are some important fixes since 4.1.1, so I think you're much better off
with 4.1.3.

Cheers,

Duncan.

[Marco Ferreira]

Hello.

Is there any way to get the incoming connection IP address?

I've browsed the web for this and I always get to getpeername() on the
file descriptor, but I'm not sure how to do this exactly.

Could someone point me on the right direction?

cheers

[Michael]

You need interceptors, check this (there might be a better resource
around though):
http://www.omniorb-support.com/pipermail/omniorb-list/2002-February/020145.html

Post by Marco Ferreira
Hello.
Is there any way to get the incoming connection IP address?
I've browsed the web for this and I always get to getpeername() on the
file descriptor, but I'm not sure how to do this exactly.
Could someone point me on the right direction?
cheers
_______________________________________________
omniORB-list mailing list
http://www.omniorb-support.com/mailman/listinfo/omniorb-list

[Jakob Happach]

Hello,

currently I'm trying to establish an omniORB for a new embedded system.

I have already made experiences with porting omniORB 3.0.4 to a older version of this operating system.

In both operating system versions the process model is a very simple one.

There are only processes (with priorities from 0 being highest to 31 being lowest priority).
Creating a posix process (posix support is available) results in creating a process with a special system predefined priority (for posix processes). There is no timeslice based scheduling (first come first serve, prority based).

In the new version of the operating system many enhancements regarding memory/filedescriptor ownership security were made.

When creating a file descriptors, the creating process is registered as owner of the handle. Thus calls of other processes using this filehandle will result in an error like EBADF.

The system handles socket descriptors the same way as file descriptors.

This means after creating a socket endpoint with the socket() call, the returned descriptor is bound to the calling process.

Trying to get omniORB 3.0.4 to operate fails as soon as the tcpSocketRendezvouser::run_undetached() call for the first incoming rope tries to call ::accept().

However there is a possibility in the operating system to forward a socket/file descriptor to another process:
- Both processes need to know each other via process handles.
- The owner process can forward a descriptor to a known receiver process. (Underlying an interprocess message is sent then from owner to receiver.)
- The receiver process has to wait for the sent descriptor.

Using a simple code like this ownership is transferred:
... GLOBAL ...
// getting sender process id from somwhere
PROCESS from;

... PROCESS A ...
pthread_t Thread;
if(pthread_create(&Thread, NULL, &ThreadB, (void *)"ThreadB") != 0)
{
std::cout << "Starting ThreadB failed!" << std::endl;
}
else
{
int iRes = 0;
from = CurrentProcess();
if ((iRes = DonateFD( pd_rendezvous, Thread))!=0)
{
std::cout << "DonateFD() is " << iRes << std::endl;
}
}

... PROCESS B ...
pd_rendezvous = ReceiveFD(from);
//pd_rendezvous is now a valid socket descriptor


As far as I am familiar with omniORB 3.0.4 I think it is a better idea to start with omniORB 4.1.3 as it is the more actual an supported version (omniORB 3.0.4 worked however perfectly for the old embedded system).

As I have already almosted adapted the omniORB 4.1.3 I now face the problem of not knowing exactly the thread model of the new omniORB good enough to know if the are different threads used for creating, using and destroying the sockets.


Currently I face the following questions:

Are different threads used for creating, using and destroying sockets?

Which threads are used for creating, using and destroying the sockets.

Where a these threads created?

Where would be the best points to forward a socket descriptor ownership?

Any hints or guesses are greatly appreciated!

Thanks a lot in advance!
Jakob

[Duncan Grisby]

Post by Jakob Happach
currently I'm trying to establish an omniORB for a new embedded system.
I have already made experiences with porting omniORB 3.0.4 to a older
version of this operating system.

[...]

Post by Jakob Happach
When creating a file descriptors, the creating process is registered
as owner of the handle. Thus calls of other processes using this
filehandle will result in an error like EBADF.

[...]

Post by Jakob Happach
As I have already almosted adapted the omniORB 4.1.3 I now face the
problem of not knowing exactly the thread model of the new omniORB
good enough to know if the are different threads used for creating,
using and destroying the sockets.

omniORB is designed with the assumption of real threads with shared
resources, including memory and sockets. It's not likely to be a small
amount of work to change that.

I assume that the "processes" on the OS share an address space? If they
can't share memory, there's no hope.

Post by Jakob Happach
Are different threads used for creating, using and destroying sockets?

Yes.

Post by Jakob Happach
Which threads are used for creating, using and destroying the sockets.

A server has one thread per listening endpoint that accepts new incoming
calls and opens sockets for the new calls. It then hands off to other
threads to process the calls. In thread pool mode, the same thread
that's listening for new calls selects / polls the open connections and
assigns incoming calls to threads. That probably wouldn't work with a
scheme where sockets are owned by particular threads, although I suppose
you could keep transferring ownership between the select thread and the
workers. In thread per connection mode, one thread is dedicated to the
connection, so that seems more appropriate for you. The system normally
still allows other threads to service interleaved calls, though, but you
can turn that off with the maxServerThreadPerConnection configuration
parameter.

Post by Jakob Happach
Where a these threads created?

They're all created by the omniAsyncInvoker. See
include/omniORB4/omniAsyncInvoker.h and
src/lib/omniORB/orbcore/invoker.cc.

Post by Jakob Happach
Where would be the best points to forward a socket descriptor ownership?

SocketCollection is responsible to selecting on sockets and notifying
threads there's something to do. giopServer::notifyRzNewConnection is
where a new connection is handled. giopServer::removeConnectionAndWorker
is the entry point for closing a connection.

Hopefully that will point you in the right direction. As I say, it's not
going to be a simple change, I'm afraid.

Cheers,

Duncan.