Add search "EXPLAIN" endpoint #17992

mpfz0r · 2024-01-18T15:19:36Z

This PR adds Warm Tier warnings for dashboard widgets and search validation.

When a dashboard is loaded or after it is updated we show an info box in widgets that retrieve data from a Warm Tier index.
When the query, time range, and streams are changed on the search page, widget edit, and dashboard override
- and the search will retrieve data from a Warm Tier we show a warning icon with information on affected time ranges.

The data tiering feature needs a way to indicate to the user that a search request will be reaching into indices that are stored in the warm tier.
Those requests are possibly slow and expensive, so we want to inform the user about this fact.
This new endpoint can be queried from the UI with a regular search request, before it is executed.
It returns which indices will be used for the search and whether those indices are stored in the warm tier or not.
Furthermore it contains the timerange info for each index, so the frontend can give the user suggestions on how to shorten the query so it does not reach the warm tiered indices.

This kind of functionality fits perfectly into a more generic "explain" endpoint.
So instead of a adding a special endpoint for just the warm tier index range info, we decided to build a more generic version which also contains a preview of the generated OS/ES query and a list of search validation errors.

Example response:

{
  "search_id": "647f0565d060431199a12e96",
  "search": {
    "queries": {
      "a1647eb6-a064-4fe6-b459-1e4267d3f659": {
        "search_types": {
          "22249f29-f042-4bd8-b745-252b00a35891": {
            "query_string": "{\"from\":0,\"size\":0,\"query\":{\"bool\":{\"must\":[{\"bool\":{\"filter\":[{\"match_all\":{\"boost\":1.0}},{\"bool\":{\"adjust_pure_negative\":true,\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},{\"range\":{\"timestamp\":{\"from\":\"2023-09-11 20:55:50.185\",\"to\":\"2024-01-18 14:49:10.185\",\"include_lower\":true,\"include_upper\":false,\"boost\":1.0}}},{\"terms\":{\"streams\":[\"63d6d52ebf9c684b3da2deb3\",\"63a5ab32e71520111ed3ce06\",\"000000000000000000000001\"],\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"track_total_hits\":2147483647,\"aggregations\":{\"agg\":{\"filters\":{\"filters\":[{\"bool\":{\"should\":[{\"exists\":{\"field\":\"source\",\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}}],\"other_bucket\":true,\"other_bucket_key\":\"_other_\"},\"aggregations\":{\"agg\":{\"terms\":{\"script\":{\"source\":\"(doc.containsKey('source') && doc['source'].size() > 0\\n? doc['source'].size() > 1\\n    ? doc['source']\\n    : String.valueOf(doc['source'].value)\\n: \\\"(Empty Value)\\\")\\n\",\"lang\":\"painless\"},\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]}}}},\"timestamp-min\":{\"min\":{\"field\":\"timestamp\"}},\"timestamp-max\":{\"max\":{\"field\":\"timestamp\"}}}}",
            "searched_index_ranges": [
              {
                "index_name": "graylog_0",
                "begin": 0,
                "end": 0,
                "is_warm_tiered": false
              },
              {
                "index_name": "bar_1512",
                "begin": 1705589036047,
                "end": 1705589284808,
                "is_warm_tiered": false
              },
              {
                "index_name": "bar_1513",
                "begin": 0,
                "end": 0,
                "is_warm_tiered": false
              },
              {
                "index_name": "bar_warm_1511",
                "begin": 1705588785906,
                "end": 1705589035782,
                "is_warm_tiered": true
              }
            ]
          },
          "5e9a9bfe-7a97-4835-86fd-896f40b20531": {
            "query_string": "{\"from\":0,\"size\":0,\"query\":{\"bool\":{\"must\":[{\"bool\":{\"filter\":[{\"match_all\":{\"boost\":1.0}},{\"bool\":{\"adjust_pure_negative\":true,\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},{\"range\":{\"timestamp\":{\"from\":\"2023-09-11 20:55:50.185\",\"to\":\"2024-01-18 14:49:10.185\",\"include_lower\":true,\"include_upper\":false,\"boost\":1.0}}},{\"terms\":{\"streams\":[\"63d6d52ebf9c684b3da2deb3\",\"63a5ab32e71520111ed3ce06\",\"000000000000000000000001\"],\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"track_total_hits\":2147483647,\"aggregations\":{\"agg\":{\"filters\":{\"filters\":[{\"bool\":{\"should\":[{\"exists\":{\"field\":\"source\",\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}}],\"other_bucket\":true,\"other_bucket_key\":\"_other_\"},\"aggregations\":{\"agg\":{\"terms\":{\"script\":{\"source\":\"(doc.containsKey('source') && doc['source'].size() > 0\\n? doc['source'].size() > 1\\n    ? doc['source']\\n    : String.valueOf(doc['source'].value)\\n: \\\"(Empty Value)\\\")\\n\",\"lang\":\"painless\"},\"size\":15,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]}}}},\"timestamp-min\":{\"min\":{\"field\":\"timestamp\"}},\"timestamp-max\":{\"max\":{\"field\":\"timestamp\"}}}}",
            "searched_index_ranges": [
              {
                "index_name": "graylog_0",
                "begin": 0,
                "end": 0,
                "is_warm_tiered": false
              }
            ]
          }
        }
      }
    }
  },
  "search_errors": [
  ]
}

refs https://github.com/Graylog2/graylog-plugin-enterprise/issues/6453

/jpd Graylog2/graylog-plugin-enterprise#6702

/nocl

coffee-squirrel · 2024-01-19T22:21:02Z

Related: #10047

The data tiering feature needs a way to indicate to the user that a search request will be reaching into indices that are stored in the warm tier. Those requests are possibly slow and expensive, so we want to inform the user about this fact. This new endpoint can be queried from the UI with a regular search request, before it is executed. It returns which indices will be used for the search and whether those indices are stored in the warm tier or not. Furthermore it contains the timerange info for each index, so the frontend can give the user suggestions on how to shorten the query so it does not reach the warm tiered indices. This kind of functionality fits perfectly into a more generic "explain" endpoint. So instead of a adding a special endpoint for just the warm tier index range info, we decided to build a more generic version which also contains a preview of the generated OS/ES query and a list of search validation errors. Example response: ``` { "search_id": "647f0565d060431199a12e96", "search": { "queries": { "a1647eb6-a064-4fe6-b459-1e4267d3f659": { "search_types": { "22249f29-f042-4bd8-b745-252b00a35891": { "query_string": "{\"from\":0,\"size\":0,\"query\":{\"bool\":{\"must\":[{\"bool\":{\"filter\":[{\"match_all\":{\"boost\":1.0}},{\"bool\":{\"adjust_pure_negative\":true,\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},{\"range\":{\"timestamp\":{\"from\":\"2023-09-11 20:55:50.185\",\"to\":\"2024-01-18 14:49:10.185\",\"include_lower\":true,\"include_upper\":false,\"boost\":1.0}}},{\"terms\":{\"streams\":[\"63d6d52ebf9c684b3da2deb3\",\"63a5ab32e71520111ed3ce06\",\"000000000000000000000001\"],\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"track_total_hits\":2147483647,\"aggregations\":{\"agg\":{\"filters\":{\"filters\":[{\"bool\":{\"should\":[{\"exists\":{\"field\":\"source\",\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}}],\"other_bucket\":true,\"other_bucket_key\":\"_other_\"},\"aggregations\":{\"agg\":{\"terms\":{\"script\":{\"source\":\"(doc.containsKey('source') && doc['source'].size() > 0\\n? doc['source'].size() > 1\\n ? doc['source']\\n : String.valueOf(doc['source'].value)\\n: \\\"(Empty Value)\\\")\\n\",\"lang\":\"painless\"},\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]}}}},\"timestamp-min\":{\"min\":{\"field\":\"timestamp\"}},\"timestamp-max\":{\"max\":{\"field\":\"timestamp\"}}}}", "searched_index_ranges": [ { "index_name": "graylog_0", "begin": 0, "end": 0, "is_warm_tiered": false }, { "index_name": "bar_1512", "begin": 1705589036047, "end": 1705589284808, "is_warm_tiered": false }, { "index_name": "bar_1513", "begin": 0, "end": 0, "is_warm_tiered": false }, { "index_name": "bar_warm_1511", "begin": 1705588785906, "end": 1705589035782, "is_warm_tiered": true } ] }, "5e9a9bfe-7a97-4835-86fd-896f40b20531": { "query_string": "{\"from\":0,\"size\":0,\"query\":{\"bool\":{\"must\":[{\"bool\":{\"filter\":[{\"match_all\":{\"boost\":1.0}},{\"bool\":{\"adjust_pure_negative\":true,\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},{\"range\":{\"timestamp\":{\"from\":\"2023-09-11 20:55:50.185\",\"to\":\"2024-01-18 14:49:10.185\",\"include_lower\":true,\"include_upper\":false,\"boost\":1.0}}},{\"terms\":{\"streams\":[\"63d6d52ebf9c684b3da2deb3\",\"63a5ab32e71520111ed3ce06\",\"000000000000000000000001\"],\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"track_total_hits\":2147483647,\"aggregations\":{\"agg\":{\"filters\":{\"filters\":[{\"bool\":{\"should\":[{\"exists\":{\"field\":\"source\",\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}}],\"other_bucket\":true,\"other_bucket_key\":\"_other_\"},\"aggregations\":{\"agg\":{\"terms\":{\"script\":{\"source\":\"(doc.containsKey('source') && doc['source'].size() > 0\\n? doc['source'].size() > 1\\n ? doc['source']\\n : String.valueOf(doc['source'].value)\\n: \\\"(Empty Value)\\\")\\n\",\"lang\":\"painless\"},\"size\":15,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"},{\"_key\":\"asc\"}]}}}},\"timestamp-min\":{\"min\":{\"field\":\"timestamp\"}},\"timestamp-max\":{\"max\":{\"field\":\"timestamp\"}}}}", "searched_index_ranges": [ { "index_name": "graylog_0", "begin": 0, "end": 0, "is_warm_tiered": false } ] } } } } }, "search_errors": [ ] } ``` Use uniqe class names for Search API The API generator cannot handle identical class names under the same endpoint. The resulting code will fail with errors like: ``` [INFO] $ /home/marco/code/graylog-project-repos/graylog2-server/graylog2-web-interface/node_modules/.bin/tsc [INFO] target/api/Search.ts(153,11): error TS6196: 'IndexRangeResult' is declared but never used. ```

…n-api

…oard

…n-api # Conflicts: # graylog-storage-elasticsearch7/src/test/java/org/graylog/storage/elasticsearch7/views/ElasticsearchBackendTest.java # graylog-storage-opensearch2/src/test/java/org/graylog/storage/opensearch2/views/OpenSearchBackendTest.java # graylog2-web-interface/src/views/components/searchbar/queryvalidation/QueryValidation.tsx

…n-api # Conflicts: # graylog-storage-elasticsearch7/src/test/java/org/graylog/storage/elasticsearch7/views/ElasticsearchBackendTest.java # graylog-storage-opensearch2/src/test/java/org/graylog/storage/opensearch2/views/OpenSearchBackendTest.java

ousmaneo

I tested the UI and it looks good. I added some suggestions.

ousmaneo · 2024-03-13T08:02:42Z

graylog2-web-interface/src/views/components/searchbar/queryvalidation/QueryValidation.tsx

@@ -182,6 +183,36 @@ const deduplicateExplanations = (explanations: Explanations | undefined): Explan
  return deduplicated;
 };

+const warmTierExplanations = (searchesWarmTier: boolean, warmTierIndices: Array<IndexRange>) => {


I think we should move the warm tier logic to its own component and use Plugin system. This component already supports it, with the views.elements.validationErrorExplanation key.

ousmaneo · 2024-03-13T08:19:28Z

graylog2-web-interface/src/views/components/widgets/Widget.tsx

@@ -240,9 +244,21 @@ const Widget = ({ id, editing, widget, title, position, onPositionsChange }: Pro
  const { config } = widget;
  const isFocused = focusedWidget?.id === id;

+  const view = useView();


Here also, we could move the logic related to the warm tier into WidgetWarTierAlert and pass the necessary props. What do you think?

I would second that, to prevent the widget component getting too cluttered.

…n-api # Conflicts: # graylog-storage-elasticsearch7/src/test/java/org/graylog/storage/elasticsearch7/views/ElasticsearchBackendTest.java # graylog-storage-opensearch2/src/test/java/org/graylog/storage/opensearch2/views/OpenSearchBackendTest.java # graylog2-server/src/main/java/org/graylog/plugins/views/search/engine/QueryEngine.java # graylog2-server/src/main/java/org/graylog/plugins/views/search/engine/SearchExecutor.java

dennisoelkers

Generally looks good. I cannot test it completely, because I am missing a data tier setup right now, but the explain functionality works fine for me. I just added some suggestions for code (structure) improvements inline.

...sticsearch7/src/main/java/org/graylog/storage/elasticsearch7/views/ElasticsearchBackend.java

dennisoelkers · 2024-03-14T09:39:52Z

...e-opensearch2/src/test/java/org/graylog/storage/opensearch2/views/OpenSearchBackendTest.java

+                .build();
+        final Search search = Search.builder().queries(ImmutableSet.of(query)).build();
+        final SearchJob job = new SearchJob("deadbeef", search, "admin", "test-node-id");
+        final GeneratedQueryContext generatedQueryContext = backend.generate(query, Set.of(), DateTimeZone.UTC);


There is ViewsUtils.createTestContext now, which makes creating a context a bit easier (and also future signature changes).

Thanks for the review 🙏
I don't see how we could pass the actual query using ViewsUtils.createTestContext()

graylog2-web-interface/src/views/bindings.tsx

graylog2-web-interface/src/views/components/widgets/Widget.tsx

dennisoelkers · 2024-03-14T09:44:41Z

graylog2-web-interface/src/views/components/widgets/Widget.tsx

@@ -240,9 +244,21 @@ const Widget = ({ id, editing, widget, title, position, onPositionsChange }: Pro
  const { config } = widget;
  const isFocused = focusedWidget?.id === id;

+  const view = useView();


I would second that, to prevent the widget component getting too cluttered.

...og2-web-interface/src/views/components/searchbar/queryvalidation/WarmTierQueryValidation.tsx

ousmaneo

The UI looks good to me.

There one comment that still need to be addressed
https://github.com/Graylog2/graylog2-server/pull/17992/files#r1524540279

ousmaneo

Tested and looks good!

mpfz0r · 2024-03-15T14:10:27Z

Merging this now.
Thanks to everyone involved. Great job 👍

mpfz0r changed the title ~~Add search explain endpoint~~ Add search "EXPLAIN" endpoint Jan 18, 2024

mpfz0r self-assigned this Jan 19, 2024

mpfz0r force-pushed the data-tiering-explain-api branch 2 times, most recently from 83999c6 to 41e480a Compare January 26, 2024 15:57

Base automatically changed from data-tiering to master January 30, 2024 11:33

mpfz0r force-pushed the data-tiering-explain-api branch from 89b5980 to fbf0c43 Compare February 7, 2024 15:43

mpfz0r force-pushed the data-tiering-explain-api branch from fbf0c43 to 197f831 Compare February 7, 2024 15:46

mpfz0r and others added 20 commits February 7, 2024 15:47

cleanup

3fafe56

Merge branch 'master' into data-tiering-explain-api

8f07e09

Merge remote-tracking branch 'origin/master' into data-tiering-explai…

69b3643

…n-api

Merge branch 'master' into data-tiering-explain-api

a5ba1c7

add index range results to search validation response

8124080

Add context for search explain and info for warm tier widgets

01490f2

Add test for SearchExplainContextProvider

a6fc027

Merge branch 'master' into data-tiering-explain-api

6d6a5ab

Merge branch 'master' into data-tiering-explain-api

12006e6

Only show warm tier info on dashboards

0b59832

Also call explain when overriding dashboard and after widget update

18e21e0

Refetch Search explain when states update, but only for created dashb…

95dcb2b

…oard

Check for warm tier indices on query validation

3b3d475

set status to warning if query validation contains warm indices

739278a

Fix spelling

414cf2a

Add tests

4477c85

Update copy

a113343

fix compile error in ElasticsearchBackendTest

ff001f6

fix compile error in OpenSearchBackendTest

b3f36ba

grotlue requested a review from ousmaneo March 7, 2024 15:37

grotlue requested a review from gally47 March 7, 2024 15:37

grotlue marked this pull request as ready for review March 7, 2024 15:38

ousmaneo requested changes Mar 13, 2024

View reviewed changes

AntonEbel and others added 4 commits March 13, 2024 14:22

Extract warm tier validation logic into own components

e053f83

Only show widget info for non edit view

9050143

Merge branch 'master' into data-tiering-explain-api

0366a85

dennisoelkers requested changes Mar 14, 2024

View reviewed changes

Address review comments

65b4fec

grotlue requested review from dennisoelkers and ousmaneo March 14, 2024 12:02

ousmaneo requested changes Mar 14, 2024

View reviewed changes

grotlue and others added 2 commits March 14, 2024 16:06

Remove obsolete function in bindings

3803bdf

fix review findings: remove code duplication

b64da2b

ousmaneo approved these changes Mar 15, 2024

View reviewed changes

dennisoelkers approved these changes Mar 15, 2024

View reviewed changes

Add warning for event definition with Warm Tier (#18591)

d102857

mpfz0r merged commit 14d6fcb into master Mar 15, 2024
6 checks passed

mpfz0r deleted the data-tiering-explain-api branch March 15, 2024 14:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add search "EXPLAIN" endpoint #17992

Add search "EXPLAIN" endpoint #17992

mpfz0r commented Jan 18, 2024 •

edited by grotlue

Loading

coffee-squirrel commented Jan 19, 2024

ousmaneo left a comment

ousmaneo Mar 13, 2024

ousmaneo Mar 13, 2024

dennisoelkers Mar 14, 2024

dennisoelkers left a comment

dennisoelkers Mar 14, 2024

mpfz0r Mar 15, 2024

dennisoelkers Mar 14, 2024

ousmaneo left a comment

ousmaneo left a comment

mpfz0r commented Mar 15, 2024

Add search "EXPLAIN" endpoint #17992

Add search "EXPLAIN" endpoint #17992

Conversation

mpfz0r commented Jan 18, 2024 • edited by grotlue Loading

coffee-squirrel commented Jan 19, 2024

ousmaneo left a comment

Choose a reason for hiding this comment

ousmaneo Mar 13, 2024

Choose a reason for hiding this comment

ousmaneo Mar 13, 2024

Choose a reason for hiding this comment

dennisoelkers Mar 14, 2024

Choose a reason for hiding this comment

dennisoelkers left a comment

Choose a reason for hiding this comment

dennisoelkers Mar 14, 2024

Choose a reason for hiding this comment

mpfz0r Mar 15, 2024

Choose a reason for hiding this comment

dennisoelkers Mar 14, 2024

Choose a reason for hiding this comment

ousmaneo left a comment

Choose a reason for hiding this comment

ousmaneo left a comment

Choose a reason for hiding this comment

mpfz0r commented Mar 15, 2024

mpfz0r commented Jan 18, 2024 •

edited by grotlue

Loading