vezel-dev / novadrop Goto Github PK

Specifically, need to check whether sending 0 in the key exchange works as expected. Also check whether the client actually expects to exchange keys at all when using the null cipher.

Hi. When runningcommand validate/pack command, I would like to be able to get the XML filename in which there is an error.
For example like this:

Unhandled exception: System.Xml.XmlException: '<', hexadecimal value 0x3C, is an invalid attribute character. Line 12, position 44.
   at System.Xml.XmlTextReaderImpl.ParseAttributeValueSlowAsync(Int32 curPos, Char quoteChar, NodeData attr)
   at System.Xml.XmlTextReaderImpl.ParseAttributesAsync()
   at System.Xml.AsyncHelper.ReturnTrueTaskWhenFinishCoreAsync(Task task)
   at System.Xml.XsdValidatingReader._ReadAsync_Read(Task`1 task)
   at Vezel.Novadrop.Commands.ValidateCommand.<>c__DisplayClass0_0.<<-ctor>b__1>d.MoveNext() in F:\novadrop-master\src\tools\dc\Commands\ValidateCommand.cs:line 51

For example, I am not sure that this attribute (on a class) is actually doing anything:

We need:

• Support for the null cipher.
• Support for the PIKE cipher.
• Support for the checksum algorithm.
• Support for TERA's packet serialization system. (Source generators?)
• A convenient connection class neatly wrapping the above + a socket.

See also:

• #1

This is a fairly major undertaking, but it will be necessary to extract navigation/collision data from the client.

Reading support is the primary goal. Creation/modification is a nice bonus.

This will cover:

• The 256-bit XOR cipher used for resource container files.
• The PIKE cipher used for network encryption.
• The (incorrect) SHA-0 implementation used for the PIKE key exchange.
• Maybe the modified CRC32 used for data center strings?

There is a problem with dismantling items in the inventory.
The problem occurs after unpacking and packing DC (also without changing an xml files).
https://www.youtube.com/watch?v=cTiMCcDT4XA

Also the problem is not present when using the repack command.

https://docs.microsoft.com/en-us/dotnet/api/system.runtime.compilerservices.skiplocalsinitattribute

This makes it difficult to e.g. use a regular expression to find warnings/errors in the output.

Tl.exe -> launcher.exe:

• 0 (csPopup() and promoPopup(%d) only; gameEvent(%d) and endPopup(%d) are understood)
• 4 (ticket; not observed; unclear how it relates to 8)
• 5 (last_svr; not observed)
• 6 (char_cnt; not observed)
• 7 (not observed)
• 8 (ticket; not observed; unclear how it relates to 4)
• 10 (getWebLinkUrl(%d,%s))

(Note that launcher.exe only ever replies to Tl.exe with the same message ID it received; it does not send its own messages.)

TERA.exe -> Tl.exe:

• 24
• 25 (potentially related to opening a Web browser; not observed)
• 26 (some kind of binary data; includes the data sent to launcher.exe as getWebLinkUrl(%d,%s) with ID 10)
• 1000 (some kind of binary data; includes the Windows user name)
• 100x and 101x (usually empty; forwarded to launcher.exe as gameEvent(%d) with ID 0)
  • 1001, 1002, 1003, 1004, 1011, and 1012 (observed)
  • 1005, 1006, 1007, 1008, 1009, 1010, 1013, and 1016 (not observed)
• 1020 (sent on normal exit; binary data including the exit reason sent to launcher.exe as endPopup(%d) with ID 0)
• 1021 (sent if the client crashes; string with exception details)
• 1022 (seems to mean "XIGNCODE3 starting"; not observed)
• 1023 (seems to mean "XIGNCODE3 started"; not observed)
• 1024 (seems to mean "XIGNCODE3 failed to start"; includes the value of GetLastError in the high bits of a 64-bit integer; not observed)
• 1025 (seems to mean "please open the customer support website"; not observed)
• 1027 (seems related to cash shop purchases; not observed)

(Tl.exe only ever acts on request messages received by TERA.exe. In response to those, it may send messages to launcher.exe to retrieve data. The reply messages sent back to TERA.exe will have a different message ID than the request message ID.)

Tl.exe -> TERA.exe:

• 27 (sent in response to message 26)

As documented in df30386.

This output format is nice for direct human interaction but is super annoying to parse from other tools. When console output is redirected, we should instead write something standard like <file>(<line>,<column>): <severity>: <message> and avoid grouping the messages under file name headings. Among other things, writing diagnostics this way integrates better with MSBuild.

It should support binary numbers (.NET 8), and it should not try to interpret the string in different bases as that can lead to wrong values. Maybe require the user to explicitly tell us the base, while defaulting to decimal.

Depends on:

• #57

Right now, we make no effort at all to deduplicate nodes and attributes. This means that data center files we produce are roughly 2x the size of official data center files.

This would have been a bigger issue in the past when TERA was 32-bit. But with the move to 64-bit, memory is effectively a non-issue. On the other hand, if people are producing multiple translated data center files for their servers, the size increase starts to add up, making distribution of the files more of an issue.

It might be good to provide an optional mode that attempts to deduplicate nodes and attributes to bring the size down at the cost of potentially much longer packing time.

Once this is done, the pack and watch commands will light up fully.

Some of this work can definitely be done mechanically.

Depends on:

• #57

This is currently an internal library used for the novadrop-scan tool. Maybe we should publish it, though?

Related to #25
I also confirm this problem.

The problem appears after repacking, even if no changes have been made to the files. There are no errors during unpacking/packing.

To be added here:

We won't ever see this from CryptoStream in Vezel.Novadrop.Formats, but it's conceivable that external users of FakePaddingCryptoTransform might call TransformFinalBlock with an array whose length is greater than the block size. We should handle that properly.

I seem to remember that they are not actually used by Tl.exe for anything (except chars_per_server), hence their omission in LauncherGameInfo, but we should confirm that.

Depends on:

• #57

See: dotnet/runtime#60639

In particular, we just need a bit of synchronization around lazy collections and table caches.

See: https://learn.microsoft.com/es-es/dotnet/csharp/language-reference/proposals/csharp-11.0/checked-user-defined-operators

This is going to run into:

• DotNetAnalyzers/StyleCopAnalyzers#3478

Also provide options on novadrop-run to apply these when the client is launched. (Or maybe do it by default?) We should have an opportunity to do so when the client makes the first launcher request since it just sits there doing nothing.

As documented in df30386. The user of the API should be able to supply their own URLs.

In particular:

• The 32-bit variant of the current version 6.
• The 2012-era version 3 (only 32-bit).

This would require rethinking of how the data center code is structured. In particular, since nodes and attributes have different layout, we'd need helper functions to access the raw fields correctly. Possibly other stuff.

This will not give the order that most people would probably expect.

It seems that communication between Tl.exe and TERA.exe uses Protocol Buffers, whereas communication between launcher.exe and Tl.exe just uses basic strings and some JSON. The latter is obviously much easier to implement, so we should go for that first - i.e. talk to Tl.exe to launch TERA.exe.

See: #17

E.g. for DownloadedResources.dat.

This will be based on past experience in Alkahest and Crux.

We should investigate whether any of these commandlets do anything interesting:

If there's anything interesting here, perhaps the launcher APIs should allow interacting with it.

See also:

• #3

This library will provide type definitions and function bindings for many of the common classes used in TERA. A few obvious examples:

• S1Context, S1CallbackHandler
• S1DataDB, S1DataQuery, S1DataNode, S1DataAttribute
• S1GameObject, S1Player, S1User
• S1ConnectionManager, S1CommandQueue
• S1ClientSocket, S1CryptSocket
• Cipher, NullCipher, PikeCipher

The goal is for this library to enable a wide array of in-memory modding scenarios.

Note: Only the unpacked r387486 client will be supported out of the box, but the library should enable people to plug in their own addresses or search patterns (e.g. for r367239).

Context: https://www.alexrp.com/p/neutralizing-protection-in-the-tera-executable

This class was always hacky and prone to race conditions. If you care about memory modification, the better answer is to just use an unpacked executable from here which has protections neutralized.

In particular:

• Describe the initial handshake and the null/PIKE ciphers.
• Describe the checksum algorithm applied to some packets.

The client doesn't seem to care either way, but we should still allow people to be good citizens and indicate the correct revision.

See

At the minimum, we should cover how launcher.exe, Tl.exe, and TERA.exe interact.