RFC: Project structure

[Ghabry]

Currently we simply convert the LDB & co. to XML and rename it to .edb & co.
This needs a solution before the editor is released to the public imo because when the first people start using it, it is too late.

Imho this could be heavily improved by splitting everything in small XML (or JSON?) files.

Though we don't want to copy from MV so I suggest something different. We split it even more:

easyrpg.project <-- Project info file
data/
|- actors/
|--- 1.xml
|--- 2.xml
|--- [...] all actors
|- common_events
|-- 1.xml
|-- 1.script
|-- 2.xml
|-- 2.script
|-- [...]
|- [...]/
|- terms.xml
|- system.xml
|- assets/
|-- CharSet
|-- ChipSet
|-- [...] everything you know
|- maps/
|-- tree.xml <-- map tree
|-- 1/ <-- Map ID
|--- map.xml <-- Describes map structure (tiles)
|--- index.xml <-- autogenerated file with event properties and start conditions
|--- 1.xml <-- Event with ID 1
|--- 1-1.script <-- Script page 1 of event 1
|--- 1-2.script 
|--- 2.xml
|--- [...]
|--- 500.xml

Cons:

• Lots of files, making operations on lots of files really slow
• Needs an (automatic generated) index at least for events because the interpreter must know which start conditions are met without parsing all files
• File system overhead (cluster size) but could be worked around by supporting zip or by merging everything together for deployment

Pros:

• Editing specific data with a text editor (or an automated script) gets easier
• Sharing single data points (e.g. 1 actor) and adding them to a different project becomes super easy
• Better colloborative editing in a repository because merge conflicts are getting less
• script is in an extra file to support a more sane scripting language later
• liblcf could abstract most of this away by adding getters/setters which automatically parse these files on access

So this is in general stuff that is only relevant for the power user and for revision control.

You can vote 👎 now ;)

[BlisterB]

Hi Ghabry ! Very glad to see that the team is moving to the Editor :).

Imho I'm not a big big fan of spliting LDB in a lot of files, it will multiply the number of syscalls at Player startup and will be a bit messy (there are already a lot of file in an RPG Maker project). But you make some good points and I'm not a core engine developper so my opinion doesn't really count.

More important, your suggestion to move to JSON format is really interesting for two reasons :

• this format (that you make me discover :p ) is really (really) lighter than xml in all field. Parsing would be quicker and files lighter.
• the main advantage of .xml is that it can be specified with DTL and Schema. But this possibility of specification doesn't seem to be very interesting in a project if we want to have the possibilty to quicly add new commands/features/mods etc.
  So XML is heavier and its main features isn't really interesting in EasyRPG : I found it not really appropriate.

Anyway, I wish you Bon Courage over my Java OCA/OCP preparations books !

[Ghabry]

Forgot to make this clear: This doesn't need a directory indexing on startup because the filename is case friendly: number.xml.

[BlisterB]

Oh please forget my sentenses about multiply files.
Your idea bring a lot of good point for the editor : the possibility to have lighter save/load (just save what have been modify, asynchronious save/load etc.). This would permit to have a really fast Editor.

RPG Maker 2000/2003 become really slow in big project. MV is just a shame : one or two second of loaging to display the database editor with a new created project. . . With a SSD in 2016, and using .JSON format my god.

[fdelapena]

I like the spare files approach, I defend this way since the first day. It should be a must to make the project collaboration easier by sharing small snippets or keep the project under some version control system. To prevent local conflicts when not using SCM, some revision number could be added (like the "number of times saved 0x5B" in lcf) or just rely on timestamp checks then checksum to, if any, show comprehensive differences compared with the local cache (git-like integration for dummies somehow? :D).

About the performance slowdown, create a large cache file (just a physfs-like dump?) and check the timestamp of spare files to keep them on sync (also useful for crash backup). When exporting a game (production status) it could be again a large package similar to the mentioned cache. To prevent too much file I/O, just use some simple archiving method, physfs-like as commented by Ghabry, keeping an abstraction should make it cheaper to maintain while keeping faster file reading.

JSON looks clearer and a bit smaller indeed, no significant performance differences but maybe a bit more human readable, yet widely supported. Cons: As BlisterBoy commented, no schema, but do we really need this? probably we don't need to do formal public specifications ;).

About naming: maybe is safer to add some semantic prefix despite the subfolder for simple spare file sharing purposes: Actors/Actor0001.xml (or .json ^^)

Well, I'm not adding nothing new, so thumbs up for all proposals so far 👍.

[MarianoGnu]

I want to add a Pro to the list:
The editor can read the files on demand, so, until you edit the character Tab of Actor 10, or find an Event Command wanting it's name the file doesn't need to be loaded.
Same happens with Chipset and souch.

[Ghabry]

Well the new winner is now obviously JSON here.

But this also depends now on a ForEachField Api. With this API the different project layouts can be quickly prototyped to find the best solution. (and then put the final solution in liblcf to ensure that Player can load it)