Rewrite and TypeScript migration

[jonbarrow]

Draft for now as

• This causes several regressions (basically all protocols are not implemented now)
• The UI is no longer implemented (the structure of data has changed quite a lot, and we're deciding if we want to redo the UI)

PR rewrites all the packet parsing and session handling essentially from scratch, and migrates the app to TypeScript.

The current version of the viewer is extremely hacked together and buggy. This is due to the fact that I originally wanted to just make a CLI tool here, and hacked the UI on top. Additionally at the time we did not have Hokaku so raw RMC messages were handled completely separately from other packets, despite having nearly identical logic

This rewrite builds on what we have learned since then, focussing on a UI-first application (rather than CLI with a hacked UI on top of it), and takes notes from our Go libraries on how to handle certain patterns

Data serialization now makes much more sense and has been standardized, as opposed to the current viewer which handles some serialization differently resulting in a hacky front end

Some key changes:

• nex-keys.txt has been deprecated. Since some games use use different key derivation methods it's not reliable to just use the pre-hashed keys anymore. Now all game server accounts are stored in the applications settings.json file as an array of objects. The objects fields are
  • platform - The platform the account is for. Unused currently, will be used in the future to support non-NEX platforms
  • username - The username. On NEX this is the string representation of the pid value
  • pid - Account PID
  • password - Plain-text password. Used to derive the other password, including any non-NEX standard ones in the future
  • password_hash_old - Pre-hashed NEX password using old key derivation
  • password_hash_new - Pre-hashed NEX password using new key derivation
• titles.json has been restructured and now contains a settings object for stream settings
  • Some data here, such as string_length_size is not currently used. It is not possible to use this information at this time, as we need to know which title the packet is for BEFORE parsing it. This is not possible to do as the title is not determined until AFTER the first SYN packet arrives. This means Rendez-Vous connections which use non-NEX settings will fail here
    • We may need to consider bringing these to global settings, however that will break cases like Rayman Legends on Wii U where both NEX and Rendez-Vous connections will be seen
    • Alternatively we can attempt to make a WireShark-style system where these packets are still shown in the UI, and the user is able to manually mark them as being for a title. The application will then re-run the parsing with this knowledge. However we still need a way to detect those packets, as currently bad packets are thrown away
  • We may also want to consider removing titles.json entirely in favor of making each title it's own JavaScript class. This way a title may define more complex settings, like a custom compression algorithm, or custom checksum algorithm (such as the custom checksums seen in WATCH_DOGS)
• All primitives have been implemented as data types
• When an RMC message is serialized to JSON, return an object whose keys are the parameter names and values are the parameters
• When a data type is serialized to JSON, the following rules apply:
  • The toJSON function MUST return an object
  • Each object MUST have both __displayTypeName and __typeName fields
  • __displayTypeName is to be used when displaying the types data in the UI. Some types, such as List, Map, etc. may reference other information about themselves in this value, making it inadequate for type referencing
  • __typeName is the types original name, and is to be used when the program needs to check what the type of the object is
  • Types which only reference a single piece of data, such as String, Bool, etc. will set their value in the __value field. This can be either a primitive value (such as the value of a Bool) OR another data type (such as the value of a AnyDataHolder)
  • Types which reference multiple pieces of data (all protocol structures, RVConnectionData, etc.) will set these is a __fields object. The keys of the __fields object are the field names, and the values are the field values
  • Data types which are Structure types store the structure version number in a __version field
  • In order all of these fields should be serialized, when applicable, as:
    • __version
    • __displayTypeName
    • __typeName
    • __value or __fields
  • Parent types have not been implemented yet, however they will likely be stored in a __parent field ABOVE the __version field
• When parsing serialized data and a __typeName is Map, this must be handled in a special way. In this case the __value field is used, and is set to an array (much like List) where each element is an object. The objects have two fields, key and value each being a data type (not a primitive). This is the only case of special handling, as ANY type may be both a key or a value which doesn't play well with JavaScript (or most languages)
• The concept of a Session has been introduced. A Session simply represents a user's play session. It parses a single packet dump to look for Connections
• A Connection is no longer identified by a string discriminator made from the address. Now it uses a combination of the address, port, and connection stream settings. This allows for titles which reuse connections with multiple streams (such as Rayman Legends) to properly split each stream into it's own connection
• Substreams are now implemented correctly
• The basis of "special protocols" has been introduced. The "special" protocols referenced in RVConnectionData are a separate server to the main secure server. When the StationURL is populated and the protocol list is not empty, clients will connect to BOTH the main and special stations. When clients make an RMC request, it checks the protocol ID and decides which to send it to. No known NEX titles use this feature, however it is partially checked now (though incomplete)

[jonbarrow]

Since this is moving to TypeScript, we may want to consider rewriting the UI in something like React or Vue. I have tagged our web developers, @hauntii and @gitlimes, because of this

[DaniElectra]

Here we are missing the updateTime field, which was added in NEX 3.6.0: https://nintendo-wiki.pretendo.network/docs/nex/protocols/ranking/#rankingrankdata-structure

We may also want to look at the original code (pre-rewrite) to ensure we don't lose documentation that may not have been ported over to the wiki

[jonbarrow]

Thank you for the update on JoinMatchmakeSessionParam. Those changes are not reflected on the wiki, I'll make a note to update that tomorrow