Nunjucks 4 roadmap / tracking ticket

[fdintino]

This issue is a draft and is still being updated

Features

• Support self variable (self variable is not defined #164)
• Support loop filtering (Support loop filtering #261)
• Support loop controls (e.g. break / continue) (Loop Controls #296)
• Recursive for loops (Recursive for loops #416)
• {% block scoped %} syntax (Any support for block "scoped" keyword from Jinja? #1118)

Bug fixes

• Make empty variables falsey (Make empty containers falsy in templates #267, Bug : you can't test a variable true-ness after an assignement from a macro #541)
• Fix nested macros and call/caller (Nested call/caller doesn't work. #664, Nested macros and Maximum call stack size exceeded #799)
• Cannot reference outer scopes from inside a {% call %} block. (Cannot reference outer scopes from inside a {% call %} block. #679)

[ArmorDarks]

Fix nested macros and call/caller (#664, #799)

Oh my, if those will be fixed, I will be obligated to buy you a beer :)

Btw, are there any plans to open Patreon or any other way to donate to maintainers?

[jgerigmeyer]

I'm guessing it relates pretty directly to #664 and #799, but #912 would be another killer bugfix to add to the list! 🍻

[atian25]

plz also support promise render

[ArmorDarks]

@atian25 right now you can simply promisify render method. It takes few lines of code.

[atian25]

yes, I had do it long time ago.

Just think about it's 2018 now, and maybe promise-base interface should be a built-in feature.

[cungminh2710]

Is a roadmap for making asynchronous calls inside macros possible ?

[fdintino]

@cungminh2710 could you elaborate a bit on what you mean?

[fdintino]

I'm finally making progress on this, and I'm encountering something that I wanted to solicit feedback on.

In short, many of these features are made much much easier to implement if we can use ES6 syntax in our compiler. As just one example, generators and for ... of greatly simplifies both the compiler code (i.e., the logic that generates the compiled template) and the resulting template js itself. The reason it's tricky to do this without those language features is because when you introduce the conditional loop syntax (e.g. {% for i in list if i % 2 == 1 %}) it is no longer sufficient to have the compiled code emit

frame.set("loop.index0", 1);
frame.set("loop.index", 2);
// ...

instead, ideally (and this is how jinja2 does it) loop should be an instance of a LoopContext that wraps an iterable passed into its constructor. As a concrete example, this template:

{% for item in list if item.index % 2 == 1 recursive %}
  {{ loop.index }}
  {% if item.children %}
    {{ loop(item.children) }}
  {% endif %}
{% endfor %}

would compile to this source (cleaned up and simplified for readability):

function root(env, context, frame, runtime, cb) {
  var lookup = runtime.memberLookup;
  function resolve(name) {
    return runtime.contextOrFrameLookup(context, frame, name);
  }
  function toStr(val) {
    return runtime.suppressValue(val, env.opts.autoescape);
  }
  function call(name, obj, args) {
    return runtime.callWrap(obj, name, context, args);
  }

  var output = "";

  try {
    var _list = resolve("list");

    function* wrapForIf(_list) {
      frame = frame.push();
      for (let _item of _list) {
        frame.set("item", _item);
        if (lookup(_item, 'index') % 2) {
          yield _item;
        }
      }
      frame = frame.pop();
    }

    function loop(iter, loopRenderFunc, depth = 0) {
      var innerOutput = "";
      var loopCtx = new runtime.LoopContext(wrapForIf(iter), loopRenderFunc, depth);

      for (let [_item, _loop] of loopCtx) {
        frame.set("item", _item);
        frame.set("loop", _loop);
        innerOutput += toStr(lookup(_item, "index"));
        if (lookup(_item, "children")) {
          innerOutput += toStr(call("loop", _loop, [lookup(_item, "children")]));
        }
      }
      return innerOutput;
    }

    output += loop(_list, loop);

    cb(null, output);
  } catch (e) {
    cb(runtime.handleError(e));
  }
}

Doing the above is possible with ES5, but it's an ugly, fragile mess. And it balloons the size of the library with a bunch of utilities and polyfills.

If I went this route, we could allow people to support old browsers with one or more of the following options:

• Recommend that people precompile their templates and hook it into their existing build tooling
• Provide built-in babel support to the precompile API for people who don't have js build tooling
• In the event that someone wants to compile a template in the browser at runtime, allow them to pass a source transformer into the Environment, so that they could use something like @babel/standalone

Thoughts?

[SalathielGenese]

Seems good so far. I upvote

[ArmorDarks]

Well... Is it the end of the Nunjucks?

[fdintino]

@ArmorDarks what do you mean?

[fdintino]

I looked into the feasibility of requiring @babel/standalone to compile nunjucks templates in-browser and determined that it would not work—it's 380k minified + gzipped; I started a project called pregenerator that can transform generators (and do other es6 transforms) while only being 88k minified and gzipped. I'd like to get it down to 60 before I think it's fully ready for use.

[ArmorDarks]

Oh, sorry. I've had an impression that the repository wasn't updated for a while.

Though, I still wonder how people able to use Nunjucks with that series of sewer scope-related bugs (#664, #799, #912) which were introduced in v3. They make the building anything but very small projects impossible.

I remember there were some movements regarding them... but it was like a few years ago...

[fdintino]

Yup. I have a v4 branch I've been working on. Those scoping issues are addressed in it. It will hopefully be at a point soon where I can release it as a public beta.

[ArmorDarks]

Ah, great to hear! It would be so nice to finally update Kotsu, since it stuck with v2 due to those bugs...

[ArmorDarks]

I'm partially missing the point why you're attempting to compile ES6 code right in the browser. I've read the post above about simplicity ES6 provides, but why would we force to compile it right inside browsers?

Today any library can provide two versions — one is totally compiled and built (usually with larger footprint), and the other one is close to the sources — es6-version with exports/imports, which can be consumed by any other tool and built according to the requirements of the project (so they can opt to include more polyfills, or target only modern browsers and have smaller js-file).

[fdintino]

Right. This would remain an option, perhaps even the default. And nunjucks-slim would not require any transpilation in-browser—it will have already been done server-side. But because nunjucks generates javascript code, and plenty of people still use nunjucks on old browsers, I felt it necessary to provide an option for those people. We will be building an esm and a legacy browser js bundle. The es6 version of the library would not include the transpilation.

[ogonkov]

@fdintino you are converting files to ES modules in your branch. It seems not break anything for 3.x branch since it's post-processed by Webpack/Babel, may be we could do it for current branch, to decrease diff of 4.x version? I could make PR.

[fdintino]

It would break some code, because it moves the files inside nunjucks to a different location. So if somebody were requiring something from nunjucks/src/environment.js (for instance) it would no longer work.

[ogonkov]

Without moving files

[fdintino]

Hmm, yes I suppose the change to rollup and ESM can be separated from the monorepo structure changes.

[dfischer0]

@fdintino any update on Nunjucks 4? Is there any chance it will come out one day?
Also, @cungminh2710 asked a while back about asynchronous support in macros and I believe you didn't get what he meant. He meant support for anything asynchronous inside a macro - such as an async filter. Right now there's a note in the documentation stating:

Important note: If you are using the asynchronous API, please be aware that you cannot do anything asynchronous inside macros. This is because macros are called like normal functions. In the future we may have a way to call a function asynchronously. If you do this now, the behavior is undefined.

I am super interested in this feature (and I have created #1320 for this.

[fdintino]

Hi @dfischer0 . I'm still making progress on nunjucks v4, but I now have a 5 month old daughter at home, which has sucked up some of the time I would otherwise be spending on this project.

The first step was to write a proof-of-concept transpiler that could convert newer ES features to ES5, but most importantly generators and async/await, while being smaller than either babel-standalone or regenerator. The result of that was pregenerator, which supports most ESNext features and has min+gz size of 88K, compared to 362K for babel-standalone and 309K for regenerator.

This was to test the feasibility of bundling a transpiler with the full browser nunjucks package, so that folks could compile templates in the browser and have them work on older browsers. Anyone who only supports modern browsers could opt for a package without the transpiler bundled, and nunjucks-slim would be unchanged, since it doesn't compile anything.

Since I'm convinced of the feasibility, I'm working on making pregenerator even smaller by replacing the pseudo-fork of babel-types with ast-types, and rewriting it in typescript.

As far as async macros, filters, loops. and the rest go: I don't think the current approach makes much sense, where we have a handful of async versions of tags. I'm partial to jinja2's approach, where the environment can be in sync or async mode. In async mode, everything is asynchronous; even otherwise synchronous filters would be wrapped in a function to make them async. Having Environment.render and Environment.renderSync seems the most idiomatic to me, for node.

[groenroos]

@fdintino Any update on Nunjucks 4? Is there something that I, and/or the wider community, could help you with? :)

[fdintino]

I've just made public the nunjucks v4 repository: https://github.com/nunjucks/nunjucks4. It supports all of the features listed in this issue description, and is correctly characterized by my comments on this issue, but it is not yet complete. There is a tracking ticket where I've listed the work that needs to be done.

I spent a fair amount of time trying to get this project to an alpha state. I welcome any and all assistance with completing the remaining work.