[sldev] message_template.msg ordering and packing : attention LLfolks

John Plevyak jplevyak at acm.org
Thu Mar 15 16:20:24 PDT 2007

On Thu, Mar 15, 2007 at 11:09:40PM +0100, Tleiades wrote:
> I don't think there are any real bugs in the message codec subsystem, it is 
> a little illogical, a little in efficient and a little over engeneered.
> >1. in the wire/binary encoding, blocks and variables are out of order
> >  with respect to the message_template.msg file
> This may be a little confusing, but it is not really a bug, a number 
> wellknown codecs have similar mismatches. E.g. PER and DER ASN.1 encoding.

I could understand if there was a reason e.g. packing or alignment, but
take ChatFromSimulator for example, the U8's are together in the template
description and scattered in the encoding.  I don't know if I would
characterize the out-of-order-ness as a bug IF the order was given by
a formula associated with the description rather than being a subtle
side-effect of the implementation as is the case here.

> >2. the ordering of blocks and variables depends the implementation of 
> >std::map
> >  and a hash function.
> The hashfunction/hash table lookup ensures cosistent numbering of 
> parameters server and client side given that the structure of the 
> message_template.msg is identical.Consistency between client side and 
> serverside layout of the template file is built into the logon sequence.

I beg to differ.  Using the ordering in the template file would ensure that.  
The hashfunction/hash table depends on the implemenation of std::map and
the order and specific entries which are PREHASHED.  The fact that it
produces a consistent mapping on the current set of platforms is 
more happenstance than design.  Perhaps it was built with the intent 
of making ordering consistent, but if so, it does it in a haphazard and 
brittle way.  I can think of many good ways of determinining an ordering, 
but iterator order of std::map over pointers to hash table entries is not 
anywhere on that list.  Just sorting them against the PREHASH order would
have been better.  Heck sort them alphabetically.

> 3. the ordering is only semi-stable
> I think this is a variation of 2
> >4. the hash function has 2 bugs in it
> The codec is inefficient in so many ways, that I don't think the performace 
> hit on this one is significant in any way, especially since the bucket 
> issue only occurs during initialization.

I don't think there is any question that a hash function which doesn't
provide a good distribution from the *domain* to the *range* is a buggy
hash function.  In this case there are 2 simple changes which don't effect
the performance of the algorithm but dramatically improve the distribution.
I would chacterize their absence as bugs, and I would suggest that 
a code review would have characterized them likewise. 

Also, gMessageStringTable.getString() occurs in such places as
LLMessageSystem::callHandler(), LLMessageSystem::addVector3(), etc. so 
I don't think it is fair to say that it is only used at initialization.

> Having said that, I do believe that the network protocol is a mess, 
> introducing one more variant - partially implemented in some packages and 
> not in others - will not improve anything.
> Currently we have a mix of XML-RPC and a hand crafted codec, adding XML 
> encoding into that will make the SL protocol even more confusing.
> The message template contains a mixture of syntax and semtic descriptions, 
> which ideally should be split.
> A lot of CPU cycles are used handling dynamic message layout, but for the 
> reasons you mention and because of login procedure starts with a 
> SecuredTemplateChecksumRequest, this flexibility cannot be taken advantage 
> of..
> Part of the messages contain variable encodings of any type, e.g. 
> essentially the same as Xml Schema ANY. Examples of such messages would be 
> ObjectUpdateCompressed and ObjectUpdate.
> IMO the protocol could be improved upon

LLSD (XML-ness) seems to be a good way to go for the lower frequency messages,
and for higher frequency messages, the message_template.msg format isn't
that bad.  If there were fewer entries (because of pulling the lower frequency
ones out) and the format was stable, one could generate serialization and
deserialization code from message_template.msg.  The result would be much faster 
and cleaner.

Does anyone actually use the "dynamic layout" aspect of the current
binary message_template.msg code?

> Tleiades
> ps. Having said that the syntax is a little inconsistent, in line 468 the 
> words "sim -> dataserver" isn't marked as a comment. Luckily the viewer 
> parser handles this inconsistency without screwing up :-)
> _______________________________________________
> Click here to unsubscribe or manage your list subscription:
> https://lists.secondlife.com/cgi-bin/mailman/listinfo/sldev

John Bradley Plevyak, PhD,   jplevyak at acm.org,   PGP KeyID: 051130BD

More information about the SLDev mailing list