"Component-level" tests of prerender_component().

[dgp1130]

In #10 we added a simple :%{component}_prerender_for_test target which allows test code to directly depend on the prerender library to verify that it works as expected. This is great for small unit tests, but unfortunately is not usable for testing interactions between the prerendered HTML and the loaded client-side JS or CSS. It also doesn't help to test that the required resources are present. A few example testing use cases which are not fulfilled here include:

• Verifying that the prerendered HTML is compatible with client-side JS.
  • Consider a component which prerenders its DOM, but then the client-side JS hydrates itself from that state. We would want a way to test that the two pieces are compatible.
• Verifying that CSS is applied to the correct elements.
  • Consider a component with complex styling which the user would like to assert or screenshot.
• Verifying that all the required resources are included.
  • Consider a component with a few images which should be confirmed to load correctly at the right locations.

All of these revolve around testing a component by prerendering it to a real page, bundling that page, then loading it into a real browser, and asserting on the page. We could conceivably make a prerender_component_test() rule which does all this work and gives a nice API for users to prerender some HTML and then use a browser driver to load that page and assert on it.

The ideal API for users might look something like this:

// my_component.ts

export function renderMyComponent(): string {
    return `<div><img src="/foo.png"></div>`;
}

# BUILD.bazel

load("@npm//@bazel/jasmine:index.bzl", "jasmine_node_test")
load("@npm//@bazel/typescript:index.bzl", "ts_library")
load(
    "@npm//rules_prerender:index.bzl",
    "prerender_component",
    "prerender_component_test_harness",
    "web_resources",
)

# Component under test.
prerender_component(
    name = "my_component",
    srcs = ["my_component.ts"],
    resources = [":resources"],
)

# Image resource required for the component.
web_resources(
    name = "resources",
    entries = {"/foo.png": "foo.png"},
)

# Generate a `ts_library()` which wraps the exported `my_component.ts` functions with the `renderer` logic.
# In this case, it will generate a `renderMyComponentHarness()` function which accepts the same parameters
# as `renderMyComponent()` but wraps the result in a complete HTML page, then bundles all the resources
# required by the component and includes them, returning a `Site` object which contains all the relevant files
# at the appropriate locations.
prerender_component_test_harness(
    name = "my_component_test_harness",
    component = ":my_component",
    testonly = True,
)

# Compile the actual test code.
ts_library(
    name = "my_component_test_lib",
    srcs = ["my_component_test.ts"],
    testonly = True,
    deps = [
        ":my_component_test_harness",
        "@npm//rules_prerender:testing",
        "@npm//@types/jasmine",
        "@npm//@types/puppeteer",
    ],
)

# Run the test.
jasmine_node_test(
    name = "my_component_test",
    data = ["@npm//puppeteer"],
    deps = [":my_component_test_lib"],
)

// my_component_test.ts

import 'jasmine';
import * as puppeteer from 'puppeteer';
import { prerenderPage, startServer, Site } from 'rules_prerender/testing';
import { renderMyComponentHarness } from './my_component_test_harness';

it('test my_component', async () => {
    const site: Site = await renderMyComponentHarness(/* params for `renderMyComponent()` */);

    const server = await startServer(site);
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.goto(`http://${server.host}:${server.port}/${site.harnessPath}`);
    const imageLoaded = await page.$eval('img[src="/foo.png"]', (img) => img.complete);
    expect(imageLoaded).toBeTrue(); // Assert image loaded successfully.

    await page.close();
    await browser.close();
    await server.shutdown();
});

Unfortunately this is quite involved to actually build. For starters it has to generate a renderMyComponentHarness() function which is roughly equivalent to the prerender_page() Starlark macro except executed entirely at runtime. As a result, this would require running the entire prerendering pipeline at runtime within a test. This includes executing the renderer to output the prerendered HTML, extracting all the JS and CSS sources, generating an entry point for each, invoking Rollup/PostCSS to bundle them together, inject those files into the resulting HTML, and then copying everything into a directory (or an in-memory object). Even once that is done, we need to implement startServer() which is effectively equivalent to web_resources_devserver() (ironically this basically already exists as useDevserver() internally, it just requires a pre-built server). This is a very confusing workflow and relies on a lot of magic in the test harness.

On the bright side I can easily avoid a dependency on any particular test framework or browser driver. If a user wanted to rewrite this test with Jest and Playwright they could do so easily. Arguably I could defer the server implementation to the user as well, but I don't think that is as practical to get away with. If we accept a hard dependency on the test framework, then we could simplify prerender_component_harness() to prerender_component_test(), which would do the same thing under the hood, but simply accept test sources as an input. Unfortunately we couldn't delegate that to a user controlled ts_library() because it needs a dependency on the harness.

As an alternative, we could leverage the existing build system to prerender the test pages at build time and then simply assert on them in the test at runtime. This might look like:

// my_component.ts

export function renderMyComponent(): string {
    return `<div><img src="/foo.png"></div>`;
}

# BUILD.bazel

load("@npm//@bazel/jasmine:index.bzl", "jasmine_node_test")
load("@npm//@bazel/typescript:index.bzl", "ts_library")
load("@npm//rules_prerender:index.bzl", "prerender_component", "prerender_pages", "web_resources")

# Component under test.
prerender_component(
    name = "my_component",
    srcs = ["my_component.ts"],
    resources = [":resources"],
)

# Image resource required for the component.
web_resources(
    name = "resources",
    entries = {"/foo.png": "foo.png"},
)

# Prerender a directory with a number of test pages to assert. This is just the regular `prerender_pages()` macro.
prerender_pages(
    name = "my_component_test_pages",
    src = "my_component_test_pages.ts",
    testonly = True,
    lib_deps = ["//@npm/rules_prerender"],
    deps = [":my_component"],
)

# Compile the actual test code.
ts_library(
    name = "my_component_test_lib",
    srcs = ["my_component_test.ts"],
    testonly = True,
    data = [":my_component_test_pages"], # Include the prerendered test pages in the test's runfiles.
    deps = [
        "@npm//rules_prerender:testing",
        "@npm//@types/jasmine",
        "@npm//@types/puppeteer",
    ],
)

# Run the test.
jasmine_node_test(
    name = "my_component_test",
    data = ["@npm//puppeteer"],
    deps = [":my_component_test_lib"],
)

// my_component_test_pages.ts

import { PrerenderResource } from 'rules_prerender';
import { renderMyComponent } from './my_component';

/** Generate test case HTML files. */
export default function*(): Generator<PrerenderResource, void, void> {
    // Generate a test case at `/loads_image.html`.
    yield PrerenderResource.of('/loads_image.html', `
        <html>
            <body>
                ${renderMyComponent()}
            </body>
        </html>
    `);

    // Generate other test cases...
}

// my_component_test.ts

import 'jasmine';
import * as puppeteer from 'puppeteer';
import { startServer } from 'rules_prerender/testing';

// References the `prerender_pages()` target.
const PRERENDERED_PAGES = `${process.env.RUNFILES}/wksp/path/to/pkg/my_component_test_pages/`;

it('test my_component', async () => {
    // Start a server which serves the prerendered test pages as a site.
    const server = await startServer(PRERENDERED_PAGES);
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.goto(`http://${server.host}:${server.port}/loads_image.html`);
    const imageLoaded = await page.$eval('img[src="/foo.png"]', (img) => img.complete);
    expect(imageLoaded).toBeTrue(); // Assert image loaded successfully.

    await page.close();
    await browser.close();
    await server.shutdown();
});

This strategy has the user explicitly generating test cases at build time using the existing prerender_pages() rule, which already handles rendering and bundling a site. I think this is cognitively much simpler because it uses infrastructure which the user is already familiar with rather than a bunch of extra magic. We also don't need to maintain a separate toolchain to replicate the behavior of prerender_pages() entirely at runtime. It also means less opportunity for divergence between the real production build and the tests being executed.

The biggest downside here is that the code which actually prerenders a page and the code which tests it are in two different files. This is a bit frustrating and makes it harder to correlate the two together. Unfortunately there is no great way to merge these two pieces into one file and doing so would probably make the system more complicated rather than less.

One interesting aspect with this strategy is that there actually isn't much work to do to support this. The only symbol which doesn't exist today is startServer(), and that could easily be implemented by the user, there's nothing rules_prerender specific there. In fact, we could probably leverage some existing library to provide that function out of the box. Users could also provide their own server if they have some special server-side functionality their application depends upon. Everything else is directly depended upon by user code, not rules_prerender. So this still supports Jest and Playwright for example.

I definitely like option 2. a lot better. It's less magical, less prone to divergence from production, and requires less infrastructure. The tricky part is documenting all this and explaining it to users. I'll also need to find (or implement) a usable server implementation for this purpose.

I think the action item here is to really just try out option 2. since it requires so little up front effort to support. If it works out well enough, then the biggest task here is just documenting and codifying the approach for real projects to use.

[dgp1130]

In 580419d, I attempted the second strategy described above and it actually worked out ok. The biggest negative is definitely that the test cases and test script are two different files. Beyond that, there is definitely a lot of boilerplate, but some more focused test helpers could probably cut it down quite a bit. Unfortunately Puppeteer loads too fast to assert the DOM before the JS is executed and the component is hydrated. I'd love to find a good way to assert on the prerendered content by itself, that would allow users to drop the separate unit test of the prerender output in many cases.

One other thought I had was to use Karma instead of Puppeteer for testing. This would likely run faster and give more convenient access to the DOM. However, I don't believe Karma has a good story for navigating to a real HTML page and then testing it. I believe Karma generally works via its own test HTML page which then loads user JavaScript onto it. There are maybe some workarounds such as parsing the test cases and stamping them to the DOM in Karma, with each test case embedded in a <template /> element to keep it unloaded until its particular test runs. However I think that is a bad idea because associated scripts and styles would be repeated for each instance and create confusing behavior. We'd need to re-run any associated JavaScript for each element, but if that script so much as defined a custom element, it would immediately fail on the second run. I think Karma could be a viable strategy if we went with option 1 above, but using option 2 definitely prefers Puppeteer, and I think I'll stick with that for the time being.

[dgp1130]

4ce3227 has reaffirmed my interest in more improvements to the testing story. I think with HydroActive being a key part of the client-side JavaScript story, it would be worth more thoroughly investigating. Following up from my previous comment with a more fleshed-out design and in light of Karma's deprecation I think the solution look something like this:

// component.prerender.tsx

import { includeScript, polyfillDeclarativeShadowDom } from '@rules_prerender/declarative_shadow_dom';
import { Template } from '@rules_prerender/preact';
import { VNode } from 'preact';

export function MyComponentPrerender({ initial = 5 }: { initial?: number }): VNode {
    return <my-component>
        <Template shadowrootmode="open">
            {polyfillDeclarativeShadowDom()}
            {includeScript('./my_component.script.mjs', import.meta)}

            <div>The count is: <span>{initial}</span>.</div>
            <button>Increment</button>
        </Template>
    </my-component>;
}

// component.script.mts

import { component } from 'hydroactive';

export const MyComponent = component('my-component', ($) => {
    const [ count, setCount ] = $.live('span', Number);

    $.listen($.query('button'), 'click', () => { setCount(count() + 1); });
});

declare global {
    interface HTMLElementTagNameMap {
        'my-component': MyComponent;
    }
}

// my_component.test.tsx

import { testCase, getEl } from '@rules_prerender/mocha';
import { renderToHtml } from '@rules_prerender/preact';
import { hydrate } from 'hydroactive';
import { MyComponentPrerender} from './component.prerender.js';
import { MyComponent } from './component.script.mjs';

describe('my_component', () => {
    describe('MyComponent', () => {
        describe('default initial value', () => {
            // Render a test case for this describe block, wraps the output in an HTML page and sets `defer-hydration` automatically.
            testCase((): SafeHtml => renderToHtml(<MyComponentPrerender />));

            it('prerenders an initial value of 5', () => {
                // Gets the prerendered element from the `testCase`.
                const el = getEl();

                expect(el.shadowRoot!.querySelector('span')!.textContent).toBe('5');
            });

            it('increments on click', () => {
                // Gets the prerendered element from the `testCase`.
                const el = getEl();

                // Hydrate the component.
                hydrate(el, MyComponent);

                // Hydration does not alter the content.
                const count = el.shadowRoot!.querySelector('span')!;
                expect(count.textContent).toBe('5');

                // Clicking the button increments the count.
                el.shadowRoot!.querySelector('button')!.click();
                expect(count.textContent).toBe('6');
            });
        });
    });
});

# BUILD.bazel

load("@rules_prerender//:index.bzl", "prerender_component", "prerender_component_test_cases")
load("@aspect_rules_jasmine//jasmine:defs.bzl", "jasmine_node_test")
load("@aspect_rules_ts//ts:defs.bzl", "ts_project")
load("@web_test_runner//wtr:defs.bzl", "wtr_test")

prerender_component(
    name = "component",
    prerender = ":prerender",
    scripts = ":scripts",
)

ts_project(
    name = "prerender",
    srcs = ["component.prerender.tsx"],
    deps = [
        "//:node_modules/@rules_prerender/preact",
        "//:node_modules/preact",
    ],
)

ts_project(
    name = "scripts",
    srcs = ["component.script.mts"],
    tsconfig = "//:tsconfig_client",
    deps = ["//:node_modules/hydroactive"],
)

ts_project(
    name = "component_test_lib",
    srcs = ["component.test.tsx"],
    tsconfig = "//:tsconfig_client",
    testonly = True,
    deps = [
        ":prerender",
        "//:node_modules/@rules_prerender/preact",
        "//:node_modules/@types/jasmine",
        "//:node_modules/@types/jasmine",
    ],
)

# Compiles and executes `component.test.tsx`, but skips all the `it` blocks. Only runs `testCase` calls and collects
# their outputs. Each test case gets wrapped in an HTML page with the result in a `<template>` element which can
# be cloned for each test. Also includes `<script>` tags to execute the associated test code at runtime.
prerender_test_cases(
    name = "component_test_cases",
    deps = [":component_test_lib"],
)

# Compiles and executes `component.test.tsx` _again_. This time however `testCase` isn't actually invoked. Instead, each
# prerendered HTML test case is executed to run the actual tests, filtered to the associated `it` blocks for each `testCase`.
wtr_test(
    name = "component_test".
    data = [":component_test_cases"],
    deps = [":component_test_lib"],
)

This works by having a single component.test.tsx file which is cross-compiled and executed at both build time and test time. The first time through in prerender_test_cases, only testCase is actually executed. The result of the render is wrapped in a "test harness" for lack of a better word. In this case, the harness looks like:

<!DOCTYPE html>
<html>
    <head>
        <title>default initial value</title>
        <meta charset="utf8">

        <!-- The component is rendered inside a template, so it can be cloned for each test. -->
        <template id="component">
            <!-- `defer-hydration` is automatically applied to the top-level rendered custom element. -->
            <my-component defer-hydration>
            </my-component>
        </template>

        <!-- Includes bundled test script. -->
        <script src="./component.test.js" type="module"></script>
    </head>
    <body></body>
</html>

Because this render happens at build-time, we can reuse prerender_pages to do all the heavy-lifting. Then at test-time, Web Test Runner is configured to load each page and inject it's own <script> tag with its setup for Jasmine. At that point, WTR will run component.test.tsx but now under different semantics. testCases will detect that it is running in a browser and convert itself into:

const template = document.head.querySelector('template#component')!;
let instance: HTMLElement | undefined;

beforeEach(() => {
    instance = template.content.cloneNode(true /* deep */);
    document.body.appendChild(instance);
});

afterEach(() => {
    instance.remove();
    instance = undefined;
});

With this, each it() block with an associated test case will run in the browser on the page with the prerendered content already present. This has a few advantages:

1. Prerendered test cases and test logic are in the same file.
2. Do not need to reimplement rendering and bundling logic at test time, can reuse prerender_pages under the hood.
3. Tree shaking can eliminate it blocks and client-side imports from the prerendered version of component.test.tsx. It can also eliminate the rendering logic from testCases in the browser version of component.test.tsx.

But there are some costs:

1. This kind of goes against the general philosophy of @rules_prerender that we don't mix Node and browser code. Here we are very much doing just that.
2. Requires DOM emulation in Node in order for component('my-component', ($) => { /* ... */ }) to not break during prerendering.
   • I wonder how far a "DOM emulation library" of window.HTMLElement = class {}; window.customElements = { define: () => {} } would get me?
3. Using tree shaking to avoid DOM emulation would significantly slow down tests.
4. Cannot name <MyComponentPrerender /> and MyComponent the same, or they will conflict in the test file.
5. Requires user to set up Web Test Runner with some probably custom configuration to load the test files and inject its framework setup.

I could potentially support another API which could make a separate component.test_cases.tsx easier to write, though it would still be quite awkward for sure. That would at least avoid negatives 1., 2., 3., and 4. at the cost of positive 1. and 3.

I'll need to think on this more and experiment. Maybe what I really want is to run browser-level tests on the prerendered components, and that's something which can happen even with a separate component.test_cases.tsx file. I'll have to experiment and see how much I like that result.