Extending (extruding?) rot.js

[dmchurch]

Hi! I'm the coding half of the team that made Deiphage for the 2024 7DRL jam, which you can see live on my repo's GH Pages site at https://dmchurch.github.io/7drl-2024/. As you can see, I've added another layer to the standard usage of rot.js - it is a classic roguelike set on a 3D tile grid 😂 I ended up creating specializations of a lot of the rot.js internal classes in order to cope with the 3D aspect of it, obviously!

I've been wanting to continue building on the idea, so I thought that now that I'm out of the frantic rush of the jam, I'd go back over all the dirty hacks I made and turn them into something that I could reuse and extend further, and that means making some fairly substantial edits to the rot.js codebase in order to support a varying number of dimensions on the majority of classes.

My question is, would you be interested in this as a potential direction (no pun intended) for the rot.js project to go in, and if so, should I work on this with the intent that it gets absorbed back into the main rot.js codebase? If so I'll work to maximize compatibility with existing projects using rot.js, and I'll also probably do some modernization of the TS code while I'm in there.

If not, that is entirely fine - it just means that I'll be creating a split for personal use, which will certainly be easier from a coding standpoint! But let me know - it'd be fun to see other rot.js projects using the stuff I've been working on, cause I can think of all sorts of ways you could use an extensible layered tile display in the browser 😄

[ondras]

Hi @dmchurch ,

thanks for the proposal! It is certainly an interesting food for thought.

I would say that while your extension to third dimension is worthwile, I am not sure it makes sense to apply it to existing rot.js objects (such as the ROT.Display class). Instead, perhaps introducing new 3d-based interfaces would be more sensible?

My intuition here is that 2d is still the main usecase for 95% of users, so it makes little sense to provide them with a more complex codebase. (The existing TS code might definitely use some modernization efforts, though.)

Apart from the the Display class, what other components of rot.js would you see fit for providing a three-dimensional API?

[dmchurch]

I am not sure it makes sense to apply it to existing rot.js objects (such as the ROT.Display class).

Oh I wholeheartedly agree that the ROT.Display class in particular wants to remain a 2D-only class! It's just far too useful to have a lightweight, simple Canvas manipulator - for example, the inventory display in Deiphage (which is coded as a popover <dialog> rather than an in-Canvas menu, since we're using a non-text tile display) uses a single ROT.Display of size 1×1 to display each item, which can then have a CSS outline applied to it when it's selected. That said, I would like to add a custom Web Component that can be used to instantiate a Display, so that you could just do

ROT.Display.registerPreset("light", {
    width: 80,
    height: 24,
    layout: "tile",
    bg: "black",
    tileWidth: 16,
    tileHeight: 16,
    tileSet: Tileset.light.img,
    tileMap: Tileset.light.tileMap,
});

and then

<rot-display preset="light" width="1" height="1" fg="red" bg="transparent">!</rot-display>

to render a single-tile Display containing a red ! tile, and perhaps

<rot-display preset="light">
    <pre>┌───┐</pre>
    <pre>│.>.│</pre>
    <pre>+.@.│</pre>
    <pre>│..f│</pre>
    <pre>└───┘</pre>
</rot-display>

to render a static image of a classic Nethack-style room with a cat, like you might want to show in a help screen; for older browsers that don't support Web Components (few and far between, these days), those tags would gracefully fall back to showing a useful text-based rendering of the desired content.

But no, in terms of 3D-esque rendering like I'm doing, I'd instead probably just import something like my Viewport class, which would act as

1. a container for multiple layers of Displays (and could contain methods to add or remove the number of layers rendered dynamically),
2. a cropping frame to display only a subset of the actual rendered Display, so that you could have a scrolling viewport without having to redraw the entire frame every time like I'm doing, as long as you're willing to set your Display to the size of your full world map, and
3. a portal to duplicate a Display being rendered elsewhere but with different settings or different cropping; think of having an inset minimap that mirrors the active game Display but using tiny tiles

Instead, perhaps introducing new 3d-based interfaces would be more sensible?

Yeah, my original inclination had been to just add a set of classes with a new 3d interface, the way I have Cellular3D, Astar3D, etc in my code. But I realized while I was working on it that there's a way to merge the 2D and 3D implementations and also allow someone to use a 4D or even 5D implementation in case, say, you wanted to track a 3D world through time, or through alternate realities, or whatever. (I could imagine using the same technique I used in Cellular3D to create z-layers that are "somewhat similar" to each other being used to instead generate alternate dimensions that are similar to but not identical to the primary, for example.)

Specifically, my thought is that instead of extending the existing rot.js code from two dimensions (with all the nested for loops to iterate through x and y) I instead reduce the code to 1 dimension. All internal coordinate-based data storage, like the internal _map of the map generator classes or the _todo/_done of the pathfinders, now just operate on a single numeric index coordinate, but they translate it back to a paired x/y coordinate to call the callbacks. They can then modify their this._dirs property to convert the x/y coordinates from DIRS into index deltas based on the total size of the map in question. Changing the internal code to use 1-dimension coordinates brings a number of advantages, like being able to use the native TypedArray classes for fast and compact map storage 😄

As for communication to/from calling code, I'd change the places that currently expect or provide separate x and y parameters to instead use a single coord parameter. On output (i.e. when calling a user-supplied callback) these would be objects of the form {x: number, y: number}, meaning that existing code could change from function(x, y, what) to function({x, y}, what) without any other changes. As for input, I'd provide some new top-level helper functions in the ROT namespace called XY and XYZ, which take numeric arguments and return coordinate objects, so existing code could take compute(x, y, callback) and change it to compute(XY(x, y), callback). Of course I'd provide a separate wrapper API that mimics the existing 2D call signatures, since that becomes pretty simple!

My intuition here is that 2d is still the main usecase for 95% of users, so it makes little sense to provide them with a more complex codebase.

I wholeheartedly agree with you here - that 95% is likely an underestimate, even! but there are really interesting things you can do with the ability to temporarily or partially extend your game to 3D. For example, in a classic Nethack-style game, you could add a second layer above the gameplay layer which only renders room walls, to give them a feeling of depth without changing the gameplay. Or you could have a dungeon floor with a hole in it and render a temporary layer below, through which you could have the player able to see what they'd be dropping onto if they step onto it. Or, for a more open-environment game than a dungeon crawler, you could use extra z-layers to, say, render trees or buildings at realistic-looking heights, which is what Door in the Woods does, and where my teammate got the idea for the parallax rendering that I used for Deiphage. I highly recommend checking out the Steam page for that game for video of what you can do with this kind of rendering when done outside the rushed context of a game jam - I'd love for that kind of look to be available to folks that don't have my level of comfort mucking around with foreign code and display algorithms!

All that said, of course, it would be an API-breaking change. Even if I made the wrapper functions the default exposed API and required consumers to choose the advanced API explicitly, the internal computation contexts require an understanding of the world-coordinate bounds in order to translate between index and (x, y) coordinate. You could make it compatible with probably 90% of existing code by assuming that the width and height passed to the first Display class constructor represent the total applicable world-bounds, but that will fail whenever someone is doing the kind of moving-viewport display that I'm doing. So, there's no question but that it'd require a major-version bump with a bit of warning documentation saying "you can PROBABLY migrate your code just by changing your import to this, but make sure to check to make sure nothing breaks".

[dmchurch]

Also worth noting, I might take some of the more genericizable classes I made for Deiphage and import them into rot.js for external use. The input layer I designed, in particular - I coded it with an eye to making it extensible, and it would give rot.js users the ability to do input handling that could natively handle keyboard/mouse/touch entry, which would be super useful if you're in a jam (pun definitely intended). And, while I very much appreciate the way that rot.js hands you tools and then gets out of the way, I could definitely see an argument for importing something like my WorldMap, which would be a batteries-included way to link generation, storage, and display that you could extend, to let you get up-and-running more quickly. And finally I could take the currently-not-recommended ROT.Engine namespace and use it to host a set of classes you could build on that would implement the Actor pattern suggested there. Just some thoughts 😄

[ondras]

Yeah, these are solid ideas, definitely a lot to chew on.

I think it would now be useful to divide the whole set of ideas into three disctinct areas:

1. declarative UI
2. more dimensions
3. misc

And try to focus on these independently. Let me share my opinions about that:

1. Declarative UI

I am a big fan of Custom Elements, so <rot-display> looks pretty interesting to me. Actually, after I finished my current 7DRL (https://github.com/ondras/roguezombies), I caught the idea of a new CustomElement-based DOM renderer for tile-based maps.

I would say that the implementation of the said <rot-display> can be done separately from the underlying ROT.Display JS instance.

Presets are yet another story. Once the <rot-display> is done, it can provide an interface for preset definition (via static methods) to be re-used by the preset attribute. Once again, there is no need for the user to interface with the low-level ROT.Display.

2. More dimensions

The idea looks simple on paper, but I am afraid you might run into issues in some algorithms that are strongly bound to the 2d realm. I would suggest trying that in a separate branch/fork and see how it goes.

Also, I like the idea of a coord single parameter, but I would argue that perhaps a number[] might be more useful when compared to {x:..., y:...}.

3. Misc

There is this weird namespacing-legacy thingy when ROT components are often available in the ROT.* namespace, but this works only if you use the pre-built JS version. Once you go for the typescript code, you import modules as necessary, naming them locally in the process. I would say that the second approach is currently the right way to go: providing individual components via ES modules, so that there is no need to usurp the ROT.Engine namespace for a particular functionality. The same obviously applies for XY/XYZ classes/objects, that would reside in a dedicated module which contains vector arithmetics etc.

Alternatively, we might want to re-use the great glMatrix project (https://glmatrix.net/) as this is a very reasonable approach for working with multi-dimensional vector data in a memory-efficient way.

[dmchurch]

1. Declarative UI

I would say that the implementation of the said <rot-display> can be done separately from the underlying ROT.Display JS instance.

Oh definitely. There's absolutely no reason to require that you use a Web Component to render a ROT.Display - declarative should always be just one way to describe a thing.

Presets are yet another story. Once the <rot-display> is done, it can provide an interface for preset definition (via static methods) to be re-used by the preset attribute. Once again, there is no need for the user to interface with the low-level ROT.Display.

You don't think so? I'd been thinking that preset could be added to the DisplayOptions interface as a simple Partial<DisplayOptions> member. I ended up instantiating a lot of Display objects for Deiphage and they all have to use new Display({...Tileset.light.getDisplayOptions(), width: x, height: y}) - I thought it could be nice to just Display.register("light", Tileset.light.getDisplayOptions()) and then be able to use it in my instantiations. That's definitely an "up to you" choice, though, so if you'd rather that only get used for the Web Component that works too!

[On rereading, I think you might have instead been more concerned with the user having to directly import and interact with the ROT.Display class when using the declarative model, and you're absolutely right they shouldn't need to! Even if preset functionality gets implemented in the low-level Display, it seems entirely reasonable to mirror the registerPreset() static method on the ROTDisplayElement class.]

2. More dimensions

The idea looks simple on paper, but I am afraid you might run into issues in some algorithms that are strongly bound to the 2d realm. I would suggest trying that in a separate branch/fork and see how it goes.

Oh yeah for sure, they wouldn't all work out nicely - anything that works with the geometry of it, like the shadowcasters, will only be happy with the dimensionality they're designed for. (I did an extremely abortive attempt to alter the PreciseShadowcasting algorithm to cope with a sphere instead of a circle before deciding that I didn't have time for that, it's a game jam, so I'm just doing the 2d shadowcast on-layer, then dropping a vertical from each visible point to wherever is visible in layers above/below.

That said, anything that only interacts with maps on a neighbor/axis basis - like pathfinding, like generation - will work just fine in whatever dimensionality, though some care needs to be taken regarding bounds; performing "x+1" on the index of a cell at the right edge of a map will result in a valid map index on the left edge of a map. For something like Cellular this can be worked around by expanding the internal map by 1 along all dimensions and forcing a boundary condition on the cells there between each generation, but otherwise it requires doing individual axis-bounds checks of the form (oldX % yStride) === (newX % yStride) (or, for bit-aligned implementations, (oldX & ~xMask) === (newX & ~xMask) - I did that for my WorldMap implementation).

I'll note also that I consider the common topology parameter to be something that would be in addition to any z-topology specification; I'd expect that any dimension after the second can be in any of the three states "no travel along this axis" (the default), "travel along this axis independently from x/y" (the 3d equivalent of "orthogonal only"), or "travel along this axis at the same time as x/y" (the 3d equivalent of diagonal movement). So, Deiphage's movement isn't "topology=10", it's "topology=8, zTopology=independent".

Also, I like the idea of a coord single parameter, but I would argue that perhaps a number[] might be more useful when compared to {x:..., y:...}.

🤦 Right, of course it would - if we're doing multidimensionality by genericizing, then limiting things to "how many coordinate letters can you come up with" is kinda silly, huh! I do like the readability that having the coordinate names gets you for the common case, though, so I think that I'd be likely to make the output side (when passing a value to a callback) a subclass of Array which has a get/set pair (or, perhaps, just a getter) for x, y, z, and w.

The one thing I do want to be cautious about is to avoid too much in the way of allocations - I'd probably specify that "the object that gets passed to the callback function will be reused, don't store the object itself, store its coordinates", and encourage that by making the type "readonly" at the Typescript level. I've had to do a lot of low-level JS optimization for my other major project, Idle Loops, and the number-one performance hit I've found for most algorithms (beyond the obvious "try to keep your big-O notation to a low factor") is avoiding allocations in critical-path functions, which means that a lot of language features like for...of become inaccessible.

3. Misc

There is this weird namespacing-legacy thingy when ROT components are often available in the ROT.* namespace, but this works only if you use the pre-built JS version. Once you go for the typescript code, you import modules as necessary, naming them locally in the process. I would say that the second approach is currently the right way to go: providing individual components via ES modules, so that there is no need to usurp the ROT.Engine namespace for a particular functionality. The same obviously applies for XY/XYZ classes/objects, that would reside in a dedicated module which contains vector arithmetics etc.

Fair enough! I was mainly thinking "namespace" in the sense of "concept space", anyway, like "where does this go in the help docs". The actual code organization matters less. (well. it matters a lot that the code BE organized. it does not matter so much HOW it is organized.)

Alternatively, we might want to re-use the great glMatrix project (https://glmatrix.net/) as this is a very reasonable approach for working with multi-dimensional vector data in a memory-efficient way.

Ooh, I haven't heard of this! I'll have to check it out but yeah, I imagine that could be quite helpful!

Shall I assume, then, that you are at least tentatively interested in bringing this functionality to rot.js? If so I'm delighted to follow your lead on how/where/what to do 😄

[ondras]

On rereading, I think you might have instead been more concerned with the user having to directly import and interact with the ROT.Display class when using the declarative model, and you're absolutely right they shouldn't need to!

Exactly, that was my point 👍

Shall I assume, then, that you are at least tentatively interested in bringing this functionality to rot.js? If so I'm delighted to follow your lead on how/where/what to do 😄

Well, yeah. Please excuse my late response; I am rather busy these days and while I am still interested in both rot.js and roguelike development, I am focusing more on other projects of mine.

As for the work organization: I would suggest making a rot.js fork where you can

1. introduce the new declarative display overlay / facade,
2. try experimenting with a multi-dimensional approach.

As for the 2), perhaps pick just one of the existing classes (pathfined, dungeon generator) and try to adjust it accordingly. Let's see how that works -- and what modifications need to be done to the public API.

[dmchurch]

Sounds good! Once my dev energy isn't wholly focused towards the continuation of our 7DRL project I'll start working on transferring some of my work over using actual proper coding technique rather than rushed jam work 😂

[dmchurch]

Ah, quick question for you. I'm working on getting a local rot.js development environment set up and testing to make sure the build gets the same results as the existing library, and I'm running into some discrepancies that I'd like to get your take on. Though first, just to check: are you capable of running a full build on your machine and getting the same results as currently checked into the repo? I've been working under that assumption but it occurred to me that it might not be accurate.

[Edited to add: I just now discovered that my fork was not up-to-date with the rebuild you in fact just did! That would have cleared up so many of my issues. Whoops!]

[ondras]

[Edited to add: I just now discovered that my fork was not up-to-date with the rebuild you in fact just did! That would have cleared up so many of my issues. Whoops!]

Yep, this was probably the source of confusion.

A full rebuild is currently a little bit problematic with respect to the auto-generated documentation, because the output html changes a bit on each run (includes timestamps). This means that an unrelated JS commit/fix influences a rather large number of files.

One approach would be to not store the documentation in the repository, running the generation task only when publishing via GitHub pages/actions. Unfortunately, I have very little experience in this field and also no time for that.

[dmchurch]

One approach would be to not store the documentation in the repository, running the generation task only when publishing via GitHub pages/actions. Unfortunately, I have very little experience in this field and also no time for that.

Heh, hear that. I can look into that on my fork as well - I actually recently had to write a custom GH action for my project to publish a manifest with checksums of all the files in the repo, just so that there would be some file on the site that would be guaranteed to change with each update, for cache-checking reasons.

One approach I've seen is to store the documentation in the same repo but on an unrelated gh-pages branch, but it'd be just as easy to run the typedoc task and publish the results to the GH page without storing them in the repo itself. Do you have a preference, between storing the generated docs in an off-branch vs keeping the live docs out of the repository entirely?

[ondras]

Do you have a preference, between storing the generated docs in an off-branch vs keeping the live docs out of the repository entirely?

No. I see both the value of using an Action (the docs being generated stuff that does not belong to the repository, in a puristic view) as well as the value of checking out a project and having the documentation at hand automatically. 🤷

[dmchurch]

On the latter note I'm already planning to look into generating an offline version of the docs for the main repo itself - strip out all the links and search metadata and it looks like they can generate in a reproducible manner, and I'd be surprised if there wasn't a way to convince typedoc to suppress those. That way the in-source docs don't change on every commit, but the online docs have live links and search functionality.

[ondras]

On the latter note I'm already planning to look into generating an offline version of the docs for the main repo itself - strip out all the links and search metadata and it looks like they can generate in a reproducible manner, and I'd be surprised if there wasn't a way to convince typedoc to suppress those. That way the in-source docs don't change on every commit, but the online docs have live links and search functionality.

Sounds like an optimal combination.

[dmchurch]

I've started some initial work on the fork! Please feel free to watch development on the rot3d branch when and as you have the time and inclination; I've added the fork as a git submodule in my post-7DRL project and I'm using Typescript's new multi-project build functionality to allow depending on (and rebuilding) the original TS sources of the rot.js code, rather than referencing the prebuilt JS modules as an NPM dependency.

Over the next week or two I'll migrate my monkey-patches and ad-hoc extensions from my 7drl repo into my rot.js fork, so that I can work with them in context and figure out where the pain points are. I'll try and keep the commit history clean on the rot.js side of things, to make reviewing the code easier; I'll let you know when I've got something substantive for you to look at.

I'll probably also be doing some general cleanup and performance improvements in the rot.js library, since performance optimization is one of my areas of expertise, and code cleanup and consistency refactoring are guilty pleasures of mine. I'd recommend that, presuming we can agree on scope and get some or all of my fork merged back into upstream, we make this a major-version change and not just a minor bump. Not because I think any of the existing API needs breaking, but just because I'd expect enough will be changing internally that any code that (like my 7drl code) does monkey-patching or uses advanced TS type manipulation on rot.js types might break in unexpected ways. I'd be much more comfortable declaring "this is a breaking change that might not actually break your code at all" than declaring "this is a non-breaking change unless you did A, B, or C in your code".

Oh and, you should feel free to mimic my setup for Deiphage if you want to try out using my fork as a submodule the way I am. I've set the rot3d branch as the main branch for my fork so that submodules using the bare git URL will fetch the branch I'm working on even without including a -b option.

[dmchurch]

I've opened #217 for you to take a look at! I'm not expecting it to actually get pulled in at the moment, so there's no rush - I'm leaving it in draft form for the time being, until I have some more substantive and useful changes to show. This should give you an idea of the type of code I write, though, as well as the degree of refactor/change I'm comfortable with.

[ondras]

we make this a major-version change and not just a minor bump

Yeah, no problems here. I am completely okay with larger changes, breaking changes, major versions etc.