893 lines
28 KiB
HTML
893 lines
28 KiB
HTML
|
<!DOCTYPE html>
|
||
|
|
<html lang="en">
|
||
|
|
|
||
|
|
<head>
|
||
|
|
<meta charset="UTF-8">
|
||
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
|
|
<title>Packet format | eAmuse API</title>
|
||
|
|
|
||
|
|
<link rel="stylesheet" href="styles.css">
|
||
|
|
|
||
|
|
</head>
|
||
|
|
|
||
|
|
<body>
|
||
|
|
<table>
|
||
|
|
<tr>
|
||
|
|
<td><a href=".">Contents</a></td>
|
||
|
|
<td><a href="./transport.html">Transport layer</a></td>
|
||
|
|
<td><a href="./packet.html">Packet format</a></td>
|
||
|
|
</tr>
|
||
|
|
</table>
|
||
|
|
|
||
|
|
<h1>Packet format</h1>
|
||
|
|
|
||
|
|
<p>eAmuse uses XML for its application layer payloads*. This XML is either verbatim, or in a custom packed binary
|
||
|
|
format.<br /><small>*Newer games use JSON, but this page is about XML.</small></p>
|
||
|
|
|
||
|
|
|
||
|
|
<h2 id="xml">The XML format</h2>
|
||
|
|
|
||
|
|
<p>Each tag that contains a value has a <code>__type</code> attribute that identifies what type it is. Array types
|
||
|
|
have a <code>__count</code> attribute indicating how many items are in the array. Binary blobs additionally have
|
||
|
|
a <code>__size</code> attribute indicating their length (this is notably not present on strings, however).</p>
|
||
|
|
<p>It is perhaps simpler to illustrate with an example, so:</p>
|
||
|
|
<pre><code><?xml version='1.0' encoding='UTF-8'?>
|
||
|
|
<call model="KFC:J:A:A:2019020600" srcid="1000" tag="b0312077">
|
||
|
|
<eventlog method="write">
|
||
|
|
<retrycnt __type="u32" />
|
||
|
|
<data>
|
||
|
|
<eventid __type="str">G_CARDED</eventid>
|
||
|
|
<eventorder __type="s32">5</eventorder>
|
||
|
|
<pcbtime __type="u64">1639669516779</pcbtime>
|
||
|
|
<gamesession __type="s64">1</gamesession>
|
||
|
|
<strdata1 __type="str" />
|
||
|
|
<strdata2 __type="str" />
|
||
|
|
<numdata1 __type="s64">1</numdata1>
|
||
|
|
<numdata2 __type="s64" />
|
||
|
|
<locationid __type="str">ea</locationid>
|
||
|
|
</data>
|
||
|
|
</eventlog>
|
||
|
|
</call></code></pre>
|
||
|
|
<p>Arrays are encoded by concatenating every value together, with spaces between them. Data types that have multiple
|
||
|
|
values, are serialized similarly.</p>
|
||
|
|
<p>Therefore, an element storing an array of <code>3u8</code> (<code>[(1, 2, 3), (4, 5, 6)]</code>) would look like
|
||
|
|
this</p>
|
||
|
|
<pre><code><demo __type="3u8" __count="2">1 2 3 4 5 6</demo></code></pre>
|
||
|
|
<p>Besides this, this is otherwise a rather standard XML.</p>
|
||
|
|
|
||
|
|
<h2 id="binary">Packed binary overview</h2>
|
||
|
|
|
||
|
|
<p>Many packets, rather than using a string-based XML format, use a custom binary packed format instead. While it
|
||
|
|
can be a little confusing, remembering that this is encoding an XML tree can make it easier to parse.</p>
|
||
|
|
<p>To start with, let's take a look at the overall structure of the packets.</p>
|
||
|
|
|
||
|
|
<table>
|
||
|
|
<thead>
|
||
|
|
<tr>
|
||
|
|
<td>0</td>
|
||
|
|
<td>1</td>
|
||
|
|
<td>2</td>
|
||
|
|
<td>3</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>5</td>
|
||
|
|
<td>6</td>
|
||
|
|
<td>7</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>9</td>
|
||
|
|
<td>10</td>
|
||
|
|
<td>11</td>
|
||
|
|
<td>12</td>
|
||
|
|
<td>13</td>
|
||
|
|
<td>14</td>
|
||
|
|
<td>15</td>
|
||
|
|
</tr>
|
||
|
|
</thead>
|
||
|
|
<tr>
|
||
|
|
<td><i>A0</i></td>
|
||
|
|
<td>C</td>
|
||
|
|
<td>E</td>
|
||
|
|
<td>~E</td>
|
||
|
|
<td colspan="4">Head length</td>
|
||
|
|
<td style="border-bottom: none" colspan="8"></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td style="border-top: none; border-bottom: none;" colspan="16">Schema definition</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td style="border-top: none;" colspan="12"></td>
|
||
|
|
<td colspan="1"><i>FF</i></td>
|
||
|
|
<td colspan="3">Align</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td colspan="4">Data length</td>
|
||
|
|
<td style="border-bottom: none" colspan="12"></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td style="border-top: none; border-bottom: none;" colspan="16">Payload</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td style="border-top: none;" colspan="13"></td>
|
||
|
|
<td colspan="3">Align</td>
|
||
|
|
</tr>
|
||
|
|
</table>
|
||
|
|
<p>Every packet starts with the magic byte <code>0xA0</code>. Following this is the content byte, the encoding byte,
|
||
|
|
and then the 2's compliment of the encoding byte.</p>
|
||
|
|
<p>Currently known possible values for the content byte are:</p>
|
||
|
|
<table>
|
||
|
|
<thead>
|
||
|
|
<tr>
|
||
|
|
<td>C</td>
|
||
|
|
<td>Content</td>
|
||
|
|
</tr>
|
||
|
|
</thead>
|
||
|
|
<tr>
|
||
|
|
<td>0x42</td>
|
||
|
|
<td>Compressed data</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x43</td>
|
||
|
|
<td>Compressed, no data</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x45</td>
|
||
|
|
<td>Decompressed data</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x46</td>
|
||
|
|
<td>Decompressed, no data</td>
|
||
|
|
</tr>
|
||
|
|
</table>
|
||
|
|
<p>Decompressed packets contain an XML string. Compressed packets are what we're interested in here.</p>
|
||
|
|
<p>The encoding flag indicates the encoding for all string types in the packet (more on those later). Possible
|
||
|
|
values are:</p>
|
||
|
|
<table>
|
||
|
|
<thead>
|
||
|
|
<tr>
|
||
|
|
<td>E</td>
|
||
|
|
<td>~E</td>
|
||
|
|
<td colspan="3">Encoding name</td>
|
||
|
|
</tr>
|
||
|
|
</thead>
|
||
|
|
|
||
|
|
<tr>
|
||
|
|
<td>0x20</td>
|
||
|
|
<td>0xDF</td>
|
||
|
|
<td>ASCII</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x40</td>
|
||
|
|
<td>0xBF</td>
|
||
|
|
<td>ISO-8859-1</td>
|
||
|
|
<td>ISO_8859-1</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x60</td>
|
||
|
|
<td>0x9F</td>
|
||
|
|
<td>EUC-JP</td>
|
||
|
|
<td>EUCJP</td>
|
||
|
|
<td>EUC_JP</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x80</td>
|
||
|
|
<td>0x7F</td>
|
||
|
|
<td>SHIFT-JIS</td>
|
||
|
|
<td>SHIFT_JIS</td>
|
||
|
|
<td>SJIS</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0xA0</td>
|
||
|
|
<td>0x5F</td>
|
||
|
|
<td>UTF-8</td>
|
||
|
|
<td>UTF8</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
</table>
|
||
|
|
<details>
|
||
|
|
<summary>Source code details</summary>
|
||
|
|
<p>The full table for these values can be found in libavs.</p>
|
||
|
|
<figure>
|
||
|
|
<img src="./images/encoding_table.png">
|
||
|
|
<figcaption><code>libavs-win32.dll:0x1006b960</code></figcaption>
|
||
|
|
</figure>
|
||
|
|
<p>A second table exists just before this on in the source, responsible for the
|
||
|
|
<code><?xml version='1.0' encoding='??'?></code> line in XML files.
|
||
|
|
</p>
|
||
|
|
<figure>
|
||
|
|
<img src="./images/xml_encoding_table.png">
|
||
|
|
<figcaption><code>libavs-win32.dll:0x1006b940</code></figcaption>
|
||
|
|
</figure>
|
||
|
|
<p>This is indexed using the following function, which maps the above encoding IDs to 1, 2, 3, 4 and 5
|
||
|
|
respectively.</p>
|
||
|
|
<pre><code>char* xml_get_encoding_name(uint encoding_id) {
|
||
|
|
return ENCODING_NAME_TABLE[((encoding_id & 0xe0) >> 5) * 4];
|
||
|
|
}</code></pre>
|
||
|
|
</details>
|
||
|
|
<p>While validating <code>~E</code> isn't technically required, it acts as a useful assertion that the packet being
|
||
|
|
parsed is valid.</p>
|
||
|
|
|
||
|
|
<h2 id="schema">The packet schema header</h2>
|
||
|
|
<p>Following the 4 byte header, is a 4 byte integer containing the length of the next part of the header (this is
|
||
|
|
technically made redundant as this structure is also terminated).</p>
|
||
|
|
<p>This part of the header defines the schema that the main payload uses.</p>
|
||
|
|
|
||
|
|
<p>A tag definition looks like:</p>
|
||
|
|
|
||
|
|
<table>
|
||
|
|
<thead>
|
||
|
|
<tr>
|
||
|
|
<td>0</td>
|
||
|
|
<td>1</td>
|
||
|
|
<td>2</td>
|
||
|
|
<td>3</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>5</td>
|
||
|
|
<td>6</td>
|
||
|
|
<td>7</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>9</td>
|
||
|
|
<td>10</td>
|
||
|
|
<td>11</td>
|
||
|
|
<td>12</td>
|
||
|
|
<td>13</td>
|
||
|
|
<td>14</td>
|
||
|
|
<td>15</td>
|
||
|
|
</tr>
|
||
|
|
</thead>
|
||
|
|
<tr>
|
||
|
|
<td>Type</td>
|
||
|
|
<td>nlen</td>
|
||
|
|
<td colspan="7">Tag name</td>
|
||
|
|
<td style="border-bottom: none" colspan="8"></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td style="border-top: none;" colspan="15">Attributes and children</td>
|
||
|
|
<td colspan="1"><i>FE</i></td>
|
||
|
|
</tr>
|
||
|
|
</table>
|
||
|
|
|
||
|
|
<p>Structure names are encoded as densely packed 6 bit values, length prefixed (<code>nlen</code>). The acceptable
|
||
|
|
alphabet is <code>0123456789:ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz</code>, and the packed values
|
||
|
|
are indecies within this alphabet.</p>
|
||
|
|
|
||
|
|
<p>The children can be a combination of either attribute names, or child tags. Attribute names are represented by
|
||
|
|
the byte <code>0x2E</code> followed by a length prefixed name as defined above. Child tags follow the above
|
||
|
|
format. Type <code>0x2E</code> must therefore be considered reserved as a possible structure type.</p>
|
||
|
|
|
||
|
|
<p>Attributes (type <code>0x2E</code>) represent a string attribute. Any other attribute must be defined as a child
|
||
|
|
tag. Is it notable that 0 children is allowable, which is how the majority of values are encoded.</p>
|
||
|
|
<p>All valid IDs, and their respective type, are listed in the following table. The bucket column here will be
|
||
|
|
used later when unpacking the main data, so we need not worry about it for now, but be warned it exists and is
|
||
|
|
possibly the least fun part of this format.</p>
|
||
|
|
|
||
|
|
<table>
|
||
|
|
<thead>
|
||
|
|
<tr>
|
||
|
|
<td>ID</td>
|
||
|
|
<td>Bytes</td>
|
||
|
|
<td>C type</td>
|
||
|
|
<td>Bucket</td>
|
||
|
|
<td colspan="2">XML names</td>
|
||
|
|
<td></td>
|
||
|
|
<td>ID</td>
|
||
|
|
<td>Bytes</td>
|
||
|
|
<td>C type</td>
|
||
|
|
<td>Bucket</td>
|
||
|
|
<td colspan="2">XML names</td>
|
||
|
|
</tr>
|
||
|
|
</thead>
|
||
|
|
<tr>
|
||
|
|
<td>0x01</td>
|
||
|
|
<td>0</td>
|
||
|
|
<td>void</td>
|
||
|
|
<td>-</td>
|
||
|
|
<td>void</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x21</td>
|
||
|
|
<td>24</td>
|
||
|
|
<td>uint64[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3u64</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x02</td>
|
||
|
|
<td>1</td>
|
||
|
|
<td>int8</td>
|
||
|
|
<td>byte</td>
|
||
|
|
<td>s8</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x22</td>
|
||
|
|
<td>12</td>
|
||
|
|
<td>float[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3f</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x03</td>
|
||
|
|
<td>1</td>
|
||
|
|
<td>uint8</td>
|
||
|
|
<td>byte</td>
|
||
|
|
<td>u8</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x23</td>
|
||
|
|
<td>24</td>
|
||
|
|
<td>double[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3d</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x04</td>
|
||
|
|
<td>2</td>
|
||
|
|
<td>int16</td>
|
||
|
|
<td>short</td>
|
||
|
|
<td>s16</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x24</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>int8[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4s8</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x05</td>
|
||
|
|
<td>2</td>
|
||
|
|
<td>uint16</td>
|
||
|
|
<td>short</td>
|
||
|
|
<td>s16</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x25</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>uint8[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4u8</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x06</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>int32</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>s32</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x26</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>int16[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4s16</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x07</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>uint32</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>u32</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x27</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>uint8[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4s16</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x08</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>int64</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>s64</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x28</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>int32[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4s32</td>
|
||
|
|
<td>vs32</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x09</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>uint64</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>u64</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x29</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>uint32[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4u32</td>
|
||
|
|
<td>vs32</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x0a</td>
|
||
|
|
<td><i>prefix</i></td>
|
||
|
|
<td>char[]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>bin</td>
|
||
|
|
<td>binary</td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x2a</td>
|
||
|
|
<td>32</td>
|
||
|
|
<td>int64[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4s64</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x0b</td>
|
||
|
|
<td><i>prefix</i></td>
|
||
|
|
<td>char[]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>str</td>
|
||
|
|
<td>string</td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x2b</td>
|
||
|
|
<td>32</td>
|
||
|
|
<td>uint64[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4u64</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x0c</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>uint8[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>ip4</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x2c</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>float[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4f</td>
|
||
|
|
<td>vf</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x0d</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>uint32</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>time</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x2d</td>
|
||
|
|
<td>32</td>
|
||
|
|
<td>double[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4d</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x0e</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>float</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>float</td>
|
||
|
|
<td>f</td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x2e</td>
|
||
|
|
<td><i>prefix</i></td>
|
||
|
|
<td>char[]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>attr</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x0f</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>double</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>double</td>
|
||
|
|
<td>d</td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x2f</td>
|
||
|
|
<td>0</td>
|
||
|
|
<td></td>
|
||
|
|
<td>-</td>
|
||
|
|
<td>array</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x10</td>
|
||
|
|
<td>2</td>
|
||
|
|
<td>int8[2]</td>
|
||
|
|
<td>short</td>
|
||
|
|
<td>2s8</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x30</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>int8[16]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>vs8</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x11</td>
|
||
|
|
<td>2</td>
|
||
|
|
<td>uint8[2]</td>
|
||
|
|
<td>short</td>
|
||
|
|
<td>2u8</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x31</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>uint8[16]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>vu8</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x12</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>int16[2]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>2s16</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x32</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>int8[8]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>vs16</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x13</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>uint16[2]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>2s16</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x33</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>uint8[8]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>vu16</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x14</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>int32[2]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>2s32</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x34</td>
|
||
|
|
<td>1</td>
|
||
|
|
<td>bool</td>
|
||
|
|
<td>byte</td>
|
||
|
|
<td>bool</td>
|
||
|
|
<td>b</td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x15</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>uint32[2]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>2u32</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x35</td>
|
||
|
|
<td>2</td>
|
||
|
|
<td>bool[2]</td>
|
||
|
|
<td>short</td>
|
||
|
|
<td>2b</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x16</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>int16[2]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>2s64</td>
|
||
|
|
<td>vs64</td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x36</td>
|
||
|
|
<td>3</td>
|
||
|
|
<td>bool[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3b</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x17</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>uint16[2]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>2u64</td>
|
||
|
|
<td>vu64</td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x37</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>bool[4]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>4b</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x18</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>float[2]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>2f</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x38</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>bool[16]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>vb</td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x19</td>
|
||
|
|
<td>16</td>
|
||
|
|
<td>double[2]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>2d</td>
|
||
|
|
<td>vd</td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x38</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x1a</td>
|
||
|
|
<td>3</td>
|
||
|
|
<td>int8[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3s8</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x39</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x1b</td>
|
||
|
|
<td>3</td>
|
||
|
|
<td>uint8[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3u8</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x3a</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x1c</td>
|
||
|
|
<td>6</td>
|
||
|
|
<td>int16[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3s16</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x3b</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x1d</td>
|
||
|
|
<td>6</td>
|
||
|
|
<td>uint16[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3s16</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x3c</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x1e</td>
|
||
|
|
<td>12</td>
|
||
|
|
<td>int32[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3s32</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x3d</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x1f</td>
|
||
|
|
<td>12</td>
|
||
|
|
<td>uint32[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3u32</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x3e</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
<tr>
|
||
|
|
<td>0x20</td>
|
||
|
|
<td>24</td>
|
||
|
|
<td>int64[3]</td>
|
||
|
|
<td>int</td>
|
||
|
|
<td>3s64</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td>0x3f</td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
<td></td>
|
||
|
|
</tr>
|
||
|
|
</table>
|
||
|
|
|
||
|
|
<p>Strings should be encoded and decoded according to the encoding specified in the packet header. Null termination is optional, however should be stripped during decoding.</p>
|
||
|
|
<p>All of these IDs are <code>& 0x3F</code>. Any value can be turned into an array by setting the 7<sup>th</sup> bit
|
||
|
|
high (<code>| 0x40</code>). Arrays of this form, in the data section, will be an aligned <code>size: u32</code>
|
||
|
|
immediately followed by <code>size</code> bytes' worth of (unaligned!) values of the unmasked type.</p>
|
||
|
|
|
||
|
|
<details>
|
||
|
|
<summary>Source code details</summary>
|
||
|
|
<p>The full table for these values can be found in libavs. This table contains the names of every tag, along
|
||
|
|
with additional information such as how many bytes that data type requires, and which parsing function
|
||
|
|
should be used.</p>
|
||
|
|
<figure>
|
||
|
|
<img src="./images/types_table.png">
|
||
|
|
<figcaption><code>libavs-win32.dll:0x100782a8</code></figcaption>
|
||
|
|
</figure>
|
||
|
|
</details>
|
||
|
|
<details>
|
||
|
|
<summary>Note about the <code>array</code> type:</summary>
|
||
|
|
<p>While I'm not totally sure, I have a suspicion this type is used internally as a pseudo-type. Trying to
|
||
|
|
identify its function as a parsable type has some obvious blockers:</p>
|
||
|
|
|
||
|
|
<p>All of the types have convenient <code>printf</code>-using helper functions that are used to emit them when
|
||
|
|
serializing XML. All except one.</p>
|
||
|
|
<img src="./images/no_array.png">
|
||
|
|
<p>If we have a look inside the function that populates node sizes (<code>libavs-win32.dll:0x1000cf00</code>),
|
||
|
|
it has an explicit case, however is the same fallback as the default case.</p>
|
||
|
|
<img src="./images/no_array_2.png">
|
||
|
|
|
||
|
|
<p>In the same function, however, we can find a second (technically first) check for the array type.</p>
|
||
|
|
<img src="./images/yes_array.png">
|
||
|
|
<p>This seems to suggest that internally arrays are represented as a normal node, with the <code>array</code>
|
||
|
|
type, however when serializing it's converted into the array types we're used to (well, will be after the
|
||
|
|
next sections) by masking 0x40 onto the contained type.</p>
|
||
|
|
<p>Also of interest from this snippet is the fact that <code>void</code>, <code>bin</code>, <code>str</code>,
|
||
|
|
and <code>attr</code> cannot be arrays. <code>void</code> and <code>attr</code> make sense, however
|
||
|
|
<code>str</code> and <code>bin</code> are more interesting. I suspect this is because konami want to be able
|
||
|
|
to preallocate the memory, which wouldn't be possible with these variable length structures.
|
||
|
|
</p>
|
||
|
|
</details>
|
||
|
|
|
||
|
|
<h2 id="data">The data section</h2>
|
||
|
|
|
||
|
|
<p>This is where all the actual packet data is. For the most part, parsing this is the easy part. We traverse our
|
||
|
|
schema, and read values out of the packet according to the value indicated in the schema. Unfortunately, konami
|
||
|
|
decided all data should be aligned very specifically, and that gaps left during alignment should be backfilled
|
||
|
|
later. This makes both reading and writing somewhat more complicated, however the system can be fairly easily
|
||
|
|
understood.</p>
|
||
|
|
<p>Firstly, we divide the payload up into 4 byte chunks. Each chunk can be allocated to either store individual
|
||
|
|
bytes, shorts, or ints (these are the buckets in the table above). When reading or writing a value, we first
|
||
|
|
check if a chunk allocated to the desired type's bucket is available and has free/as-yet-unread space within it.
|
||
|
|
If so, we will store/read our data to/from there. If there is no such chunk, we claim the next unclaimed chunk
|
||
|
|
for our bucket.</p>
|
||
|
|
<p>For example, imagine we write the sequence <code>byte, int, byte, short, byte, int, short</code>. The final output should look like:</p>
|
||
|
|
|
||
|
|
<table>
|
||
|
|
<thead>
|
||
|
|
<tr>
|
||
|
|
<td>0</td>
|
||
|
|
<td>1</td>
|
||
|
|
<td>2</td>
|
||
|
|
<td>3</td>
|
||
|
|
<td>4</td>
|
||
|
|
<td>5</td>
|
||
|
|
<td>6</td>
|
||
|
|
<td>7</td>
|
||
|
|
<td>8</td>
|
||
|
|
<td>9</td>
|
||
|
|
<td>10</td>
|
||
|
|
<td>11</td>
|
||
|
|
<td>12</td>
|
||
|
|
<td>13</td>
|
||
|
|
<td>14</td>
|
||
|
|
<td>15</td>
|
||
|
|
</tr>
|
||
|
|
</thead>
|
||
|
|
<tr>
|
||
|
|
<td>byte</td>
|
||
|
|
<td>byte</td>
|
||
|
|
<td>byte</td>
|
||
|
|
<td></td>
|
||
|
|
<td colspan="4">int</td>
|
||
|
|
<td colspan="2">short</td>
|
||
|
|
<td colspan="2">short</td>
|
||
|
|
<td colspan="4">int</td>
|
||
|
|
</tr>
|
||
|
|
</table>
|
||
|
|
|
||
|
|
<p>While this might seem a silly system compared to just not aligning values, it is at least possible to intuit that it helps reduce wasted space. It should be noted that any variable-length structure, such as a string or an array, claims all chunks it encroaches on for the <code>int</code> bucket, disallowing the storage of bytes or shorts within them.</p>
|
||
|
|
|
||
|
|
<details>
|
||
|
|
<summary>Implementing a packer</summary>
|
||
|
|
<p>While the intuitive way to understand the packing algorithm is via chunks and buckets, a far more efficient implementation can be made that uses three pointers. Rather than try to explain in words, hopefully this python implementation should suffice as explanation:<pre><code>class Packer:
|
||
|
|
def __init__(self, offset=0):
|
||
|
|
self._word_cursor = offset
|
||
|
|
self._short_cursor = offset
|
||
|
|
self._byte_cursor = offset
|
||
|
|
self._boundary = offset % 4
|
||
|
|
|
||
|
|
def _next_block(self):
|
||
|
|
self._word_cursor += 4
|
||
|
|
return self._word_cursor - 4
|
||
|
|
|
||
|
|
def request_allocation(self, size):
|
||
|
|
if size == 0:
|
||
|
|
return self._word_cursor
|
||
|
|
elif size == 1:
|
||
|
|
if self._byte_cursor % 4 == self._boundary:
|
||
|
|
self._byte_cursor = self._next_block() + 1
|
||
|
|
else:
|
||
|
|
self._byte_cursor += 1
|
||
|
|
return self._byte_cursor - 1
|
||
|
|
elif size == 2:
|
||
|
|
if self._short_cursor % 4 == self._boundary:
|
||
|
|
self._short_cursor = self._next_block() + 2
|
||
|
|
else:
|
||
|
|
self._short_cursor += 2
|
||
|
|
return self._short_cursor - 2
|
||
|
|
else:
|
||
|
|
old_cursor = self._word_cursor
|
||
|
|
for _ in range(math.ceil(size / 4)):
|
||
|
|
self._word_cursor += 4
|
||
|
|
return old_cursor
|
||
|
|
|
||
|
|
def notify_skipped(self, no_bytes):
|
||
|
|
for _ in range(math.ceil(no_bytes / 4)):
|
||
|
|
self.request_allocation(4)</code></pre></p>
|
||
|
|
</details>
|
||
|
|
|
||
|
|
</body>
|
||
|
|
|
||
|
|
</html>
|