How I built search-as-you-type UX with Sphinx's built-in client-side JS search tools

[kaycebasques]

On pigweed.dev I managed to hack together a search-as-you-type UX using only Sphinx's built-in client-side JS search logic. As you type stuff into the search query textbox, you see inline results under your query:

366233420-5dcd1b9f-2ad6-4457-aca6-acb4286fe262.webm

I have heard that other theme maintainers have wanted something similar but couldn't figure out how to get it working. So I thought it might be helpful to share the general outline of the implementation here. I also want to make the upstream Sphinx maintainers aware of this work because it requires extending searchtools.js a bit beyond its assumed usage.

I'm in the process of upstreaming the pigweed.dev implementation into pydata-sphinx-theme. Here's the PR: pydata/pydata-sphinx-theme#2093

Implementation

Extend basic theme

It seems like searchtools.js (the file containing upstream Sphinx's client-side JS search logic) is only available if your theme extends basic.

Update layout to always load searchtools.js

Update your main HTML layout template to (almost) always load searchtools.js and the other search-related files.

  {% if pagename == 'search' %}
    {# Search tools are already loaded on search page. Don't load them twice. #}
  {% else %}
    {# Load Sphinx's built-in search tools so that our custom inline search
       experience can work on any page. #}
    <script src="{{ pathto('_static/searchtools.js', 1) | e }}"></script>
    <script src="{{ pathto('_static/language_data.js', 1) | e }}"></script>
    <script src="{{ pathto('searchindex.js', 1) | e }}"></script>
  {% endif %}

Listen for searches, kick off the search, make sure the results container exists

The actual search logic is completely offloaded to searchtools.js. You just need to kick off the search and make sure that searchtools.js has somewhere to render the results. There is some complexity around making sure that the search result URLs are correct. The full WIP impl for pydata-sphinx-theme is below.

/*******************************************************************************
 * Inline search results (search-as-you-type)
 *
 * Immediately displays search results under the search query textbox.
 *
 * The search is conducted by Sphinx's built-in search tools (searchtools.js).
 * Usually searchtools.js is only available on /search.html but
 * pydata-sphinx-theme (PST) has been modified to load searchtools.js on every
 * page. After the user types something into PST's search query textbox,
 * searchtools.js executes the search and populates the results into
 * the #search-results container. searchtools.js expects the results container
 * to have that exact ID.
 */
var setupSearchAsYouType = () => {
  if (!DOCUMENTATION_OPTIONS.search_as_you_type) {  // In pydata-sphinx-theme we're making it a configurable option
    return;
  }

  // Don't interfere with the default search UX on /search.html.
  if (window.location.pathname.endsWith("/search.html")) {
    return;
  }

  // Bail if the Search class is not available. Search-as-you-type is
  // impossible without that class. layout.html should ensure that
  // searchtools.js loads.
  //
  // Search class is defined in upstream Sphinx:
  // https://github.com/sphinx-doc/sphinx/blob/master/sphinx/themes/basic/static/searchtools.js#L181
  if (!Search) {
    return;
  }

  // Destroy the previous search container and create a new one.
  resetSearchAsYouTypeResults();
  let timeoutId = null;
  let lastQuery = "";
  const searchInput = document.querySelector(
    "#pst-search-dialog input[name=q]",
  );

  // Initiate searches whenever the user types stuff in the search modal textbox.
  searchInput.addEventListener("keyup", () => {
    const query = searchInput.value;

    // Don't search when there's nothing in the query textbox.
    if (query === "") {
      resetSearchAsYouTypeResults(); // Remove previous results.
      return;
    }

    // Don't search if there is no detectable change between
    // the last query and the current query. E.g. the user presses
    // Tab to start navigating the search results.
    if (query === lastQuery) {
      return;
    }

    // The user has changed the search query. Delete the old results
    // and start setting up the new container.
    resetSearchAsYouTypeResults();

    // Debounce so that the search only starts when the user stops typing.
    const delay_ms = 300;
    lastQuery = query;
    if (timeoutId) {
      window.clearTimeout(timeoutId);
    }
    timeoutId = window.setTimeout(() => {
      Search.performSearch(query);
      document.querySelector("#search-results").classList.remove("empty");
      timeoutId = null;
    }, delay_ms);
  });
};

// Delete the old search results container (if it exists) and set up a new one.
//
// There is some complexity around ensuring that the search results links are
// correct because we're extending searchtools.js past its assumed usage.
// Sphinx assumes that searches are only executed from /search.html and
// therefore it assumes that all search results links should be relative to
// the root directory of the website. In our case the search can now execute
// from any page of the website so we must fix the relative URLs that
// searchtools.js generates.
var resetSearchAsYouTypeResults = () => {
  if (!DOCUMENTATION_OPTIONS.search_as_you_type) {
    return;
  }
  // If a search-as-you-type results container was previously added,
  // remove it now.
  let results = document.querySelector("#search-results");
  if (results) {
    results.remove();
  }

  // Create a new search-as-you-type results container.
  results = document.createElement("section");
  results.classList.add("search-results");
  results.classList.add("empty");
  // IMPORTANT: The search results container MUST have this exact ID.
  // searchtools.js is hardcoded to populate into the node with this ID.
  results.id = "search-results";
  let modal = document.querySelector("#pst-search-dialog");
  modal.appendChild(results);

  // Get the relative path back to the root of the website.
  const root =
    "URL_ROOT" in DOCUMENTATION_OPTIONS
      ? DOCUMENTATION_OPTIONS.URL_ROOT // Sphinx v6 and earlier
      : document.documentElement.dataset.content_root; // Sphinx v7 and later

  // As Sphinx populates the search results, this observer makes sure that
  // each URL is correct (i.e. doesn't 404).
  const linkObserver = new MutationObserver(() => {
    const links = Array.from(
      document.querySelectorAll("#search-results .search a"),
    );
    // Check every link every time because the timing of when new results are
    // added is unpredictable and it's not an expensive operation.
    links.forEach((link) => {
      // Don't use the link.href getter because the browser computes the href
      // as a full URL. We need the relative URL that Sphinx generates.
      const href = link.getAttribute("href");
      if (href.startsWith(root)) {
        // No work needed. The root has already been prepended to the href.
        return;
      }
      link.href = `${root}${href}`;
    });
  });

  // The node that linkObserver watches doesn't exist until the user types
  // something into the search textbox. This second observer (resultsObserver)
  // just waits for #search-results to exist and then registers
  // linkObserver on it.
  let isObserved = false;
  const resultsObserver = new MutationObserver(() => {
    if (isObserved) {
      return;
    }
    const container = document.querySelector("#search-results .search");
    if (!container) {
      return;
    }
    linkObserver.observe(container, { childList: true });
    isObserved = true;
  });
  resultsObserver.observe(results, { childList: true });
};

Suggestions for upstream Sphinx

• A lot of the implementation complexity revolves around making sure that the search results URLs are correct. The Search class in searchtools.js assumes that searches are only executed from /search.html whereas now it's possible to search from any page on the site. I think the Search class constructor should accept a new argument indicating what page the search is currently executing from, and then Search class itself should take care of making sure that the relative URLs are correct.

• searchtools.js assumes that a #search-results node exists and populates the results into that container. It might be better to have Search accept another new parameter indicating where the search results should be populated into.

• The most flexible option would probably be some kind of pub/sub API where upstream Sphinx just blasts off events while it conducts the search and then the theme maintainer has complete freedom to render the search results however they see fit based off the underlying data.

[liborjelinek]

It's so cool and creative! If you don't mind, I've added your built-in search customization as an outstanding example to my older post about implementing search to Sphinx themes at https://documatt.com/blog/25/sphinx-builtin-search-theme/.