Packet format

eAmuse uses XML for its application layer payloads*. This XML is either verbatim, or in a custom packed binary format.
*Newer games use JSON, but this page is about XML.

The XML format

Each tag that contains a value has a __type attribute that identifies what type it is. Array types have a __count attribute indicating how many items are in the array. Binary blobs additionally have a __size attribute indicating their length (this is notably not present on strings, however).

It is perhaps simpler to illustrate with an example, so:

<?xml version='1.0' encoding='UTF-8'?>
<call model="KFC:J:A:A:2019020600" srcid="1000" tag="b0312077">
    <eventlog method="write">
        <retrycnt __type="u32" />
        <data>
            <eventid __type="str">G_CARDED</eventid>
            <eventorder __type="s32">5</eventorder>
            <pcbtime __type="u64">1639669516779</pcbtime>
            <gamesession __type="s64">1</gamesession>
            <strdata1 __type="str" />
            <strdata2 __type="str" />
            <numdata1 __type="s64">1</numdata1>
            <numdata2 __type="s64" />
            <locationid __type="str">ea</locationid>
        </data>
    </eventlog>
</call>

Arrays are encoded by concatenating every value together, with spaces between them. Data types that have multiple values, are serialized similarly.

Therefore, an element storing an array of 3u8 ([(1, 2, 3), (4, 5, 6)]) would look like this

<demo __type="3u8" __count="2">1 2 3 4 5 6</demo>

Besides this, this is otherwise a rather standard XML.

Packed binary overview

Many packets, rather than using a string-based XML format, use a custom binary packed format instead. While it can be a little confusing, remembering that this is encoding an XML tree can make it easier to parse.

To start with, let's take a look at the overall structure of the packets.

Every packet starts with the magic byte 0xA0. Following this is the content byte, the encoding byte, and then the 2's compliment of the encoding byte.

Currently known possible values for the content byte are:

Decompressed packets contain an XML string. Compressed packets are what we're interested in here.

The encoding flag indicates the encoding for all string types in the packet (more on those later). Possible values are:

Source code details

The full table for these values can be found in libavs.

A second table exists just before this on in the source, responsible for the <?xml version='1.0' encoding='??'?> line in XML files.

This is indexed using the following function, which maps the above encoding IDs to 1, 2, 3, 4 and 5 respectively.

char* xml_get_encoding_name(uint encoding_id) {
    return ENCODING_NAME_TABLE[((encoding_id & 0xe0) >> 5) * 4];
}

While validating ~E isn't technically required, it acts as a useful assertion that the packet being parsed is valid.

The packet schema header

Following the 4 byte header, is a 4 byte integer containing the length of the next part of the header (this is technically made redundant as this structure is also terminated).

This part of the header defines the schema that the main payload uses.

A tag definition looks like:

Structure names are encoded as densely packed 6 bit values, length prefixed (nlen). The acceptable alphabet is 0123456789:ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz, and the packed values are indecies within this alphabet.

The children can be a combination of either attribute names, or child tags. Attribute names are represented by the byte 0x2E followed by a length prefixed name as defined above. Child tags follow the above format. Type 0x2E must therefore be considered reserved as a possible structure type.

Attributes (type 0x2E) represent a string attribute. Any other attribute must be defined as a child tag. Is it notable that 0 children is allowable, which is how the majority of values are encoded.

All valid IDs, and their respective type, are listed in the following table. The bucket column here will be used later when unpacking the main data, so we need not worry about it for now, but be warned it exists and is possibly the least fun part of this format.

Strings should be encoded and decoded according to the encoding specified in the packet header. Null termination is optional, however should be stripped during decoding.

All of these IDs are & 0x3F. Any value can be turned into an array by setting the 7^th bit high (| 0x40). Arrays of this form, in the data section, will be an aligned size: u32 immediately followed by size bytes' worth of (unaligned!) values of the unmasked type.

Source code details

The full table for these values can be found in libavs. This table contains the names of every tag, along with additional information such as how many bytes that data type requires, and which parsing function should be used.

Note about the array type:

While I'm not totally sure, I have a suspicion this type is used internally as a pseudo-type. Trying to identify its function as a parsable type has some obvious blockers:

All of the types have convenient printf-using helper functions that are used to emit them when serializing XML. All except one.

If we have a look inside the function that populates node sizes (libavs-win32.dll:0x1000cf00), it has an explicit case, however is the same fallback as the default case.

In the same function, however, we can find a second (technically first) check for the array type.

This seems to suggest that internally arrays are represented as a normal node, with the array type, however when serializing it's converted into the array types we're used to (well, will be after the next sections) by masking 0x40 onto the contained type.

Also of interest from this snippet is the fact that void, bin, str, and attr cannot be arrays. void and attr make sense, however str and bin are more interesting. I suspect this is because konami want to be able to preallocate the memory, which wouldn't be possible with these variable length structures.

The data section

This is where all the actual packet data is. For the most part, parsing this is the easy part. We traverse our schema, and read values out of the packet according to the value indicated in the schema. Unfortunately, konami decided all data should be aligned very specifically, and that gaps left during alignment should be backfilled later. This makes both reading and writing somewhat more complicated, however the system can be fairly easily understood.

Firstly, we divide the payload up into 4 byte chunks. Each chunk can be allocated to either store individual bytes, shorts, or ints (these are the buckets in the table above). When reading or writing a value, we first check if a chunk allocated to the desired type's bucket is available and has free/as-yet-unread space within it. If so, we will store/read our data to/from there. If there is no such chunk, we claim the next unclaimed chunk for our bucket.

For example, imagine we write the sequence byte, int, byte, short, byte, int, short. The final output should look like:

While this might seem a silly system compared to just not aligning values, it is at least possible to intuit that it helps reduce wasted space. It should be noted that any variable-length structure, such as a string or an array, claims all chunks it encroaches on for the int bucket, disallowing the storage of bytes or shorts within them.

class Packer:
    def __init__(self, offset=0):
        self._word_cursor = offset
        self._short_cursor = offset
        self._byte_cursor = offset
        self._boundary = offset % 4

    def _next_block(self):
        self._word_cursor += 4
        return self._word_cursor - 4

    def request_allocation(self, size):
        if size == 0:
            return self._word_cursor
        elif size == 1:
            if self._byte_cursor % 4 == self._boundary:
                self._byte_cursor = self._next_block() + 1
            else:
                self._byte_cursor += 1
            return self._byte_cursor - 1
        elif size == 2:
            if self._short_cursor % 4 == self._boundary:
                self._short_cursor = self._next_block() + 2
            else:
                self._short_cursor += 2
            return self._short_cursor - 2
        else:
            old_cursor = self._word_cursor
            for _ in range(math.ceil(size / 4)):
                self._word_cursor += 4
            return old_cursor

    def notify_skipped(self, no_bytes):
        for _ in range(math.ceil(no_bytes / 4)):
            self.request_allocation(4)