{:/} in the toolbar.
+The following tutorials guide you through using OpenSearch Assistant in OpenSearch Dashboards. OpenSearch Assistant can be viewed in full frame or in the sidebar. The default view is in the right sidebar. To view the assistant in the left sidebar or in full frame, select the {::nomarkdown}{:/} icon in the toolbar and choose the preferred option.
### Start a conversation
diff --git a/_dashboards/management/index-patterns.md b/_dashboards/management/index-patterns.md
index 590a9675a2..37baa210e9 100644
--- a/_dashboards/management/index-patterns.md
+++ b/_dashboards/management/index-patterns.md
@@ -56,7 +56,7 @@ An example of step 1 is shown in the following image. Note that the index patter
Once the index pattern has been created, you can view the mapping of the matching indexes. Within the table, you can see the list of fields, along with their data type and properties. An example is shown in the following image.
-
+
## Next steps
diff --git a/_dashboards/management/multi-data-sources.md b/_dashboards/management/multi-data-sources.md
index 0447348648..dd66101f80 100644
--- a/_dashboards/management/multi-data-sources.md
+++ b/_dashboards/management/multi-data-sources.md
@@ -3,7 +3,7 @@ layout: default
title: Configuring and using multiple data sources
parent: Data sources
nav_order: 10
-redirect_from:
+redirect_from:
- /dashboards/discover/multi-data-sources/
---
@@ -11,23 +11,22 @@ redirect_from:
You can ingest, process, and analyze data from multiple data sources in OpenSearch Dashboards. You configure the data sources in the **Dashboards Management** > **Data sources** app, as shown in the following image.
-
## Getting started
-The following tutorial guides you through configuring and using multiple data sources.
+The following tutorial guides you through configuring and using multiple data sources.
### Step 1: Modify the YAML file settings
To use multiple data sources, you must enable the `data_source.enabled` setting. It is disabled by default. To enable multiple data sources:
1. Open your local copy of the OpenSearch Dashboards configuration file, `opensearch_dashboards.yml`. If you don't have a copy, [`opensearch_dashboards.yml`](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/config/opensearch_dashboards.yml) is available on GitHub.
-2. Set `data_source.enabled:` to `true` and save the YAML file.
+2. Set `data_source.enabled:` to `true` and save the YAML file.
3. Restart the OpenSearch Dashboards container.
4. Verify that the configuration settings were configured properly by connecting to OpenSearch Dashboards and viewing the **Dashboards Management** navigation menu. **Data sources** appears in the sidebar. You'll see a view similar to the following image.
- 
+
### Step 2: Create a new data source connection
@@ -36,16 +35,17 @@ A data source connection specifies the parameters needed to connect to a data so
To create a new data source connection:
1. From the OpenSearch Dashboards main menu, select **Dashboards Management** > **Data sources** > **Create data source connection**.
-2. Add the required information to each field to configure **Connection Details** and **Authentication Method**.
-
+
+2. Add the required information to each field to configure the **Connection Details** and **Authentication Method**.
+
- Under **Connection Details**, enter a title and endpoint URL. For this tutorial, use the URL `http://localhost:5601/app/management/opensearch-dashboards/dataSources`. Entering a description is optional.
- Under **Authentication Method**, select an authentication method from the dropdown list. Once an authentication method is selected, the applicable fields for that method appear. You can then enter the required details. The authentication method options are:
- - **No authentication**: No authentication is used to connect to the data source.
- - **Username & Password**: A basic username and password are used to connect to the data source.
- - **AWS SigV4**: An AWS Signature Version 4 authenticating request is used to connect to the data source. AWS Signature Version 4 requires an access key and a secret key.
- - For AWS Signature Version 4 authentication, first specify the **Region**. Next, select the OpenSearch service in the **Service Name** list. The options are **Amazon OpenSearch Service** and **Amazon OpenSearch Serverless**. Last, enter the **Access Key** and **Secret Key** for authorization.
-
+ - **No authentication**: No authentication is used to connect to the data source.
+ - **Username & Password**: A basic username and password are used to connect to the data source.
+ - **AWS SigV4**: An AWS Signature Version 4 authenticating request is used to connect to the data source. AWS Signature Version 4 requires an access key and a secret key.
+ - For AWS Signature Version 4 authentication, first specify the **Region**. Next, select the OpenSearch service from the **Service Name** list. The options are **Amazon OpenSearch Service** and **Amazon OpenSearch Serverless**. Last, enter the **Access Key** and **Secret Key** for authorization.
+
For information about available AWS Regions for AWS accounts, see [Available Regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions). For more information about AWS Signature Version 4 authentication requests, see [Authenticating Requests (AWS Signature Version 4)](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html).
{: .note}
@@ -58,12 +58,11 @@ To create a new data source connection:
- To make changes to the data source connection, select a connection in the list on the **Data Sources** main page. The **Connection Details** window opens.
- To make changes to **Connection Details**, edit one or both of the **Title** and **Description** fields and select **Save changes** in the lower-right corner of the screen. You can also cancel changes here. To change the **Authentication Method**, choose a different authentication method, enter your credentials (if applicable), and then select **Save changes** in the lower-right corner of the screen. The changes are saved.
-
+
- When **Username & Password** is the selected authentication method, you can update the password by choosing **Update stored password** next to the **Password** field. In the pop-up window, enter a new password in the first field and then enter it again in the second field to confirm. Select **Update stored password** in the pop-up window. The new password is saved. Select **Test connection** to confirm that the connection is valid.
-
- When **AWS SigV4** is the selected authentication method, you can update the credentials by selecting **Update stored AWS credential**. In the pop-up window, enter a new access key in the first field and a new secret key in the second field. Select **Update stored AWS credential** in the pop-up window. The new credentials are saved. Select **Test connection** in the upper-right corner of the screen to confirm that the connection is valid.
-5. Delete the data source connection by selecting the check box to the left of the title and then choosing **Delete 1 connection**. Selecting multiple check boxes for multiple connections is supported. Alternatively, select the trash can icon ({::nomarkdown}
{:/}).
+5. Delete the data source connection by selecting the check box to the left of the title and then choosing **Delete 1 connection**. Selecting multiple check boxes for multiple connections is supported. Alternatively, select the {::nomarkdown}{:/} icon.
An example data source connection screen is shown in the following image.
@@ -71,7 +70,7 @@ An example data source connection screen is shown in the following image.
### Selecting multiple data sources through the Dev Tools console
-Alternatively, you can select multiple data sources through the [Dev Tools]({{site.url}}{{site.baseurl}}/dashboards/dev-tools/index-dev/) console. This option provides for working with a broader range of data and gaining deeper insight into your code and applications.
+Alternatively, you can select multiple data sources through the [Dev Tools]({{site.url}}{{site.baseurl}}/dashboards/dev-tools/index-dev/) console. This option allows you to work with a broader range of data and gaining a deeper understanding of your code and applications.
Watch the following 10-second video to see it in action.
@@ -79,7 +78,7 @@ Watch the following 10-second video to see it in action.
To select a data source through the Dev Tools console, follow these steps:
-1. Locate your copy of `opensearch_dashboards.yml` and open it in the editor of your choice.
+1. Locate your copy of `opensearch_dashboards.yml` and open it in the editor of your choice.
2. Set `data_source.enabled` to `true`.
3. Connect to OpenSearch Dashboards and select **Dev Tools** in the menu.
4. Enter the following query in the editor pane of the **Console** and then select the play button:
@@ -93,19 +92,55 @@ To select a data source through the Dev Tools console, follow these steps:
6. Repeat the preceding steps for each data source you want to select.
### Upload saved objects to a dashboard from connected data sources
-To upload saved objects from connected data sources to a dashboard with multiple data sources, export them as an NDJSON file from the data source's **Saved object management** page. Then upload the file to the dashboard's **Saved object management** page. This method can make it easier to transfer saved objects between dashboards. The following 20-second video shows this feature in action.
+To upload saved objects from connected data sources to a dashboard with multiple data sources, export them as an NDJSON file from the data source's **Saved object management** page. Then upload the file to the dashboard's **Saved object management** page. This method can simplify the transfer of saved objects between dashboards. The following 20-second video shows this feature in action.
{: .img-fluid}
+#### Import saved objects from a connected data source
+
Follow these steps to import saved objects from a connected data source:
-1. Locate your `opensearch_dashboards.yml` file and open it in your preferred text editor.
+1. Locate your `opensearch_dashboards.yml` file and open it in your preferred text editor.
2. Set `data_source.enabled` to `true`.
3. Connect to OpenSearch Dashboards and go to **Dashboards Management** > **Saved objects**.
4. Select **Import** > **Select file** and upload the file acquired from the connected data source.
5. Choose the appropriate **Data source** from the dropdown menu, set your **Conflict management** option, and then select the **Import** button.
+### Show or hide authentication methods for multiple data sources
+Introduced 2.13
+{: .label .label-purple }
+
+A feature flag in your `opensearch_dashboards.yml` file allows you to show or hide authentication methods within the `data_source` plugin. The following example setting, shown in a 10-second demo, hides the authentication method for `AWSSigV4`.
+
+````
+# Set enabled to false to hide the authentication method from multiple data source in OpenSearch Dashboards.
+# If this setting is commented out, then all three options will be available in OpenSearch Dashboards.
+# The default value will be considered as true.
+data_source.authTypes:
+ NoAuthentication:
+ enabled: true
+ UsernamePassword:
+ enabled: true
+ AWSSigV4:
+ enabled: false
+````
+
+
{: .img-fluid}
+
+### Hide the local cluster option for multiple data sources
+Introduced 2.13
+{: .label .label-purple }
+
+A feature flag in your `opensearch_dashboards.yml` file allows you to hide the local cluster option within the `data_source` plugin. This option hides the local cluster from the data source dropdown menu and index creation page, which is ideal for environments with or without a local OpenSearch cluster. The following example setting, shown in a 20-second demo, hides the local cluster.
+
+````
+# hide local cluster in the data source dropdown and index pattern creation page.
+data_source.hideLocalCluster: true
+````
+
+
{: .img-fluid}
+
## Next steps
Once you've configured your multiple data sources, you can start exploring that data. See the following resources to learn more:
@@ -120,5 +155,5 @@ Once you've configured your multiple data sources, you can start exploring that
This feature has some limitations:
* The multiple data sources feature is supported for index-pattern-based visualizations only.
-* The visualization types Time Series Visual Builder (TSVB), Vega and Vega-Lite, and timeline are not supported.
-* External plugins, such as Gantt chart, and non-visualization plugins, such as the developer console, are not supported.
+* The Time Series Visual Builder (TSVB) and timeline visualization types are not supported.
+* External plugins, such as `gantt-chart`, and non-visualization plugins are not supported.
diff --git a/_dashboards/visualize/vega.md b/_dashboards/visualize/vega.md
new file mode 100644
index 0000000000..7764d583a6
--- /dev/null
+++ b/_dashboards/visualize/vega.md
@@ -0,0 +1,192 @@
+---
+layout: default
+title: Using Vega
+parent: Building data visualizations
+nav_order: 45
+---
+
+# Using Vega
+
+[Vega](https://vega.github.io/vega/) and [Vega-Lite](https://vega.github.io/vega-lite/) are open-source, declarative language visualization tools that you can use to create custom data visualizations with your OpenSearch data and [Vega Data](https://vega.github.io/vega/docs/data/). These tools are ideal for advanced users comfortable with writing OpenSearch queries directly. Enable the `vis_type_vega` plugin in your `opensearch_dashboards.yml` file to write your [Vega specifications](https://vega.github.io/vega/docs/specification/) in either JSON or [HJSON](https://hjson.github.io/) format or to specify one or more OpenSearch queries within your Vega specification. By default, the plugin is set to `true`. The configuration is shown in the following example. For configuration details, refer to the `vis_type_vega` [README](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/plugins/vis_type_vega/README.md).
+
+```
+vis_type_vega.enabled: true
+```
+
+The following image shows a custom Vega map created in OpenSearch.
+
+
+
+## Querying from multiple data sources
+
+If you have configured [multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/) in OpenSearch Dashboards, you can use Vega to query those data sources. Within your Vega specification, add the `data_source_name` field under the `url` property to target a specific data source by name. By default, queries use data from the local cluster. You can assign individual `data_source_name` values to each OpenSearch query within your Vega specification. This allows you to query multiple indexes across different data sources in a single visualization.
+
+The following is an example Vega specification with `Demo US Cluster` as the specified `data_source_name`:
+
+```
+{
+ $schema: https://vega.github.io/schema/vega/v5.json
+ config: {
+ kibana: {type: "map", latitude: 25, longitude: -70, zoom: 3}
+ }
+ data: [
+ {
+ name: table
+ url: {
+ index: opensearch_dashboards_sample_data_flights
+ // This OpenSearchQuery will query from the Demo US Cluster datasource
+ data_source_name: Demo US Cluster
+ %context%: true
+ // Uncomment to enable time filtering
+ // %timefield%: timestamp
+ body: {
+ size: 0
+ aggs: {
+ origins: {
+ terms: {field: "OriginAirportID", size: 10000}
+ aggs: {
+ originLocation: {
+ top_hits: {
+ size: 1
+ _source: {
+ includes: ["OriginLocation", "Origin"]
+ }
+ }
+ }
+ distinations: {
+ terms: {field: "DestAirportID", size: 10000}
+ aggs: {
+ destLocation: {
+ top_hits: {
+ size: 1
+ _source: {
+ includes: ["DestLocation"]
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ format: {property: "aggregations.origins.buckets"}
+ transform: [
+ {
+ type: geopoint
+ projection: projection
+ fields: [
+ originLocation.hits.hits[0]._source.OriginLocation.lon
+ originLocation.hits.hits[0]._source.OriginLocation.lat
+ ]
+ }
+ ]
+ }
+ {
+ name: selectedDatum
+ on: [
+ {trigger: "!selected", remove: true}
+ {trigger: "selected", insert: "selected"}
+ ]
+ }
+ ]
+ signals: [
+ {
+ name: selected
+ value: null
+ on: [
+ {events: "@airport:mouseover", update: "datum"}
+ {events: "@airport:mouseout", update: "null"}
+ ]
+ }
+ ]
+ scales: [
+ {
+ name: airportSize
+ type: linear
+ domain: {data: "table", field: "doc_count"}
+ range: [
+ {signal: "zoom*zoom*0.2+1"}
+ {signal: "zoom*zoom*10+1"}
+ ]
+ }
+ ]
+ marks: [
+ {
+ type: group
+ from: {
+ facet: {
+ name: facetedDatum
+ data: selectedDatum
+ field: distinations.buckets
+ }
+ }
+ data: [
+ {
+ name: facetDatumElems
+ source: facetedDatum
+ transform: [
+ {
+ type: geopoint
+ projection: projection
+ fields: [
+ destLocation.hits.hits[0]._source.DestLocation.lon
+ destLocation.hits.hits[0]._source.DestLocation.lat
+ ]
+ }
+ {type: "formula", expr: "{x:parent.x, y:parent.y}", as: "source"}
+ {type: "formula", expr: "{x:datum.x, y:datum.y}", as: "target"}
+ {type: "linkpath", shape: "diagonal"}
+ ]
+ }
+ ]
+ scales: [
+ {
+ name: lineThickness
+ type: log
+ clamp: true
+ range: [1, 8]
+ }
+ {
+ name: lineOpacity
+ type: log
+ clamp: true
+ range: [0.2, 0.8]
+ }
+ ]
+ marks: [
+ {
+ from: {data: "facetDatumElems"}
+ type: path
+ interactive: false
+ encode: {
+ update: {
+ path: {field: "path"}
+ stroke: {value: "black"}
+ strokeWidth: {scale: "lineThickness", field: "doc_count"}
+ strokeOpacity: {scale: "lineOpacity", field: "doc_count"}
+ }
+ }
+ }
+ ]
+ }
+ {
+ name: airport
+ type: symbol
+ from: {data: "table"}
+ encode: {
+ update: {
+ size: {scale: "airportSize", field: "doc_count"}
+ xc: {signal: "datum.x"}
+ yc: {signal: "datum.y"}
+ tooltip: {
+ signal: "{title: datum.originLocation.hits.hits[0]._source.Origin + ' (' + datum.key + ')', connnections: length(datum.distinations.buckets), flights: datum.doc_count}"
+ }
+ }
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
diff --git a/_data-prepper/common-use-cases/trace-analytics.md b/_data-prepper/common-use-cases/trace-analytics.md
index 1f6c3b7cc4..033830351a 100644
--- a/_data-prepper/common-use-cases/trace-analytics.md
+++ b/_data-prepper/common-use-cases/trace-analytics.md
@@ -38,9 +38,9 @@ The [OpenTelemetry source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/c
There are three processors for the trace analytics feature:
-* *otel_traces_raw* - The *otel_traces_raw* processor receives a collection of [span](https://github.com/opensearch-project/data-prepper/blob/fa65e9efb3f8d6a404a1ab1875f21ce85e5c5a6d/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records from [*otel-trace-source*]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-trace/), and performs stateful processing, extraction, and completion of trace-group-related fields.
-* *otel_traces_group* - The *otel_traces_group* processor fills in the missing trace-group-related fields in the collection of [span](https://github.com/opensearch-project/data-prepper/blob/298e7931aa3b26130048ac3bde260e066857df54/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the OpenSearch backend.
-* *service_map_stateful* – The *service_map_stateful* processor performs the required preprocessing for trace data and builds metadata to display the `service-map` dashboards.
+* otel_traces_raw -- The *otel_traces_raw* processor receives a collection of [span](https://github.com/opensearch-project/data-prepper/blob/fa65e9efb3f8d6a404a1ab1875f21ce85e5c5a6d/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records from [*otel-trace-source*]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-trace-source/), and performs stateful processing, extraction, and completion of trace-group-related fields.
+* otel_traces_group -- The *otel_traces_group* processor fills in the missing trace-group-related fields in the collection of [span](https://github.com/opensearch-project/data-prepper/blob/298e7931aa3b26130048ac3bde260e066857df54/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the OpenSearch backend.
+* service_map_stateful -- The *service_map_stateful* processor performs the required preprocessing for trace data and builds metadata to display the `service-map` dashboards.
### OpenSearch sink
@@ -49,8 +49,8 @@ OpenSearch provides a generic sink that writes data to OpenSearch as the destina
The sink provides specific configurations for the trace analytics feature. These configurations allow the sink to use indexes and index templates specific to trace analytics. The following OpenSearch indexes are specific to trace analytics:
-* *otel-v1-apm-span* – The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) processor.
-* *otel-v1-apm-service-map* – The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map-stateful/) processor.
+* otel-v1-apm-span –- The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) processor.
+* otel-v1-apm-service-map –- The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map-stateful/) processor.
## Trace tuning
@@ -374,4 +374,4 @@ Starting with Data Prepper version 1.4, trace processing uses Data Prepper's eve
* `otel_traces_group` replaces `otel_traces_group_prepper` for event-based spans.
In Data Prepper version 2.0, `otel_traces_source` will only output events. Data Prepper version 2.0 also removes `otel_traces_raw_prepper` and `otel_traces_group_prepper` entirely. To migrate to Data Prepper version 2.0, you can configure your trace pipeline using the event model.
-
\ No newline at end of file
+
diff --git a/_data-prepper/pipelines/configuration/sources/otel-trace.md b/_data-prepper/pipelines/configuration/sources/otel-trace-source.md
similarity index 89%
rename from _data-prepper/pipelines/configuration/sources/otel-trace.md
rename to _data-prepper/pipelines/configuration/sources/otel-trace-source.md
index 4b17647768..137592bbe8 100644
--- a/_data-prepper/pipelines/configuration/sources/otel-trace.md
+++ b/_data-prepper/pipelines/configuration/sources/otel-trace-source.md
@@ -1,22 +1,22 @@
---
layout: default
-title: otel_trace_source source
+title: otel_trace_source
parent: Sources
grand_parent: Pipelines
nav_order: 15
+redirect_from:
+ - /data-prepper/pipelines/configuration/sources/otel-trace/
---
-# otel_trace source
+# otel_trace_source
-## Overview
-
-The `otel_trace` source is a source for the OpenTelemetry Collector. The following table describes options you can use to configure the `otel_trace` source.
+`otel_trace_source` is a source for the OpenTelemetry Collector. The following table describes options you can use to configure the `otel_trace_source` source.
Option | Required | Type | Description
:--- | :--- | :--- | :---
-port | No | Integer | The port that the `otel_trace` source runs on. Default value is `21890`.
+port | No | Integer | The port that the `otel_trace_source` source runs on. Default value is `21890`.
request_timeout | No | Integer | The request timeout, in milliseconds. Default value is `10000`.
health_check_service | No | Boolean | Enables a gRPC health check service under `grpc.health.v1/Health/Check`. Default value is `false`.
unauthenticated_health_check | No | Boolean | Determines whether or not authentication is required on the health check endpoint. Data Prepper ignores this option if no authentication is defined. Default value is `false`.
@@ -35,6 +35,8 @@ authentication | No | Object | An authentication configuration. By default, an u
## Metrics
+The 'otel_trace_source' source includes the following metrics.
+
### Counters
- `requestTimeouts`: Measures the total number of requests that time out.
@@ -50,4 +52,4 @@ authentication | No | Object | An authentication configuration. By default, an u
### Distribution summaries
-- `payloadSize`: Measures the incoming request payload size distribution in bytes.
\ No newline at end of file
+- `payloadSize`: Measures the incoming request payload size distribution in bytes.
diff --git a/_im-plugin/reindex-data.md b/_im-plugin/reindex-data.md
index 2e3288087a..a766589b84 100644
--- a/_im-plugin/reindex-data.md
+++ b/_im-plugin/reindex-data.md
@@ -91,6 +91,12 @@ Options | Valid values | Description | Required
`socket_timeout` | Time Unit | The wait time for socket reads (default 30s). | No
`connect_timeout` | Time Unit | The wait time for remote connection timeouts (default 30s). | No
+The following table lists the retry policy cluster settings.
+
+Setting | Description | Default value
+:--- | :---
+`reindex.remote.retry.initial_backoff` | The initial backoff time for retries. Subsequent retries will follow exponential backoff based on the initial backoff time. | 500 ms
+`reindex.remote.retry.max_count` | The maximum number of retry attempts. | 15
## Reindex a subset of documents
diff --git a/_ingest-pipelines/processors/index-processors.md b/_ingest-pipelines/processors/index-processors.md
index fb71e90d01..60fcac82e2 100644
--- a/_ingest-pipelines/processors/index-processors.md
+++ b/_ingest-pipelines/processors/index-processors.md
@@ -59,6 +59,7 @@ Processor type | Description
`sort` | Sorts the elements of an array in ascending or descending order.
`sparse_encoding` | Generates a sparse vector/token and weights from text fields for neural sparse search using sparse retrieval.
`split` | Splits a field into an array using a separator character.
+`text_chunking` | Splits long documents into smaller chunks.
`text_embedding` | Generates vector embeddings from text fields for semantic search.
`text_image_embedding` | Generates combined vector embeddings from text and image fields for multimodal neural search.
`trim` | Removes leading and trailing white space from a string field.
diff --git a/_ingest-pipelines/processors/text-chunking.md b/_ingest-pipelines/processors/text-chunking.md
new file mode 100644
index 0000000000..e9ff55b210
--- /dev/null
+++ b/_ingest-pipelines/processors/text-chunking.md
@@ -0,0 +1,315 @@
+---
+layout: default
+title: Text chunking
+parent: Ingest processors
+nav_order: 250
+---
+
+# Text chunking processor
+
+The `text_chunking` processor splits a long document into shorter passages. The processor supports the following algorithms for text splitting:
+
+- [`fixed_token_length`](#fixed-token-length-algorithm): Splits text into passages of the specified size.
+- [`delimiter`](#delimiter-algorithm): Splits text into passages on a delimiter.
+
+The following is the syntax for the `text_chunking` processor:
+
+```json
+{
+ "text_chunking": {
+ "field_map": {
+ "- `connector.pre_process.cohere.embedding` for [Cohere](https://cohere.com/) embedding models
- `connector.pre_process.openai.embedding` for [OpenAI](https://openai.com/) embedding models
- `connector.pre_process.default.embedding`, which you can use to preprocess documents in neural search requests so that they are in the format that ML Commons can process with the default preprocessor (OpenSearch 2.11 or later). For more information, see [built-in functions](#built-in-pre--and-post-processing-functions). | -| `post_process_function` | String | Optional. A built-in or custom Painless script used to post-process the model output data. OpenSearch provides the following built-in post-process functions that you can call directly:
- `connector.pre_process.cohere.embedding` for [Cohere text embedding models](https://docs.cohere.com/reference/embed)
- `connector.pre_process.openai.embedding` for [OpenAI text embedding models](https://platform.openai.com/docs/api-reference/embeddings)
- `connector.post_process.default.embedding`, which you can use to post-process documents in the model response so that they are in the format that neural search expects (OpenSearch 2.11 or later). For more information, see [built-in functions](#built-in-pre--and-post-processing-functions). | +| Field | Data type | Is required | Description | +|:------------------------|:------------|:------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `name` | String | Yes | The name of the connector. | +| `description` | String | Yes | A description of the connector. | +| `version` | Integer | Yes | The version of the connector. | +| `protocol` | String | Yes | The protocol for the connection. For AWS services such as Amazon SageMaker and Amazon Bedrock, use `aws_sigv4`. For all other services, use `http`. | +| `parameters` | JSON object | Yes | The default connector parameters, including `endpoint` and `model`. Any parameters indicated in this field can be overridden by parameters specified in a predict request. | +| `credential` | JSON object | Yes | Defines any credential variables required to connect to your chosen endpoint. ML Commons uses **AES/GCM/NoPadding** symmetric encryption to encrypt your credentials. When the connection to the cluster first starts, OpenSearch creates a random 32-byte encryption key that persists in OpenSearch's system index. Therefore, you do not need to manually set the encryption key. | +| `actions` | JSON array | Yes | Defines what actions can run within the connector. If you're an administrator creating a connection, add the [blueprint]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/) for your desired connection. | +| `backend_roles` | JSON array | Yes | A list of OpenSearch backend roles. For more information about setting up backend roles, see [Assigning backend roles to users]({{site.url}}{{site.baseurl}}/ml-commons-plugin/model-access-control#assigning-backend-roles-to-users). | +| `access_mode` | String | Yes | Sets the access mode for the model, either `public`, `restricted`, or `private`. Default is `private`. For more information about `access_mode`, see [Model groups]({{site.url}}{{site.baseurl}}/ml-commons-plugin/model-access-control#model-groups). | +| `add_all_backend_roles` | Boolean | Yes | When set to `true`, adds all `backend_roles` to the access list, which only a user with admin permissions can adjust. When set to `false`, non-admins can add `backend_roles`. | +| `client_config` | JSON object | No | The client configuration object, which provides settings that control the behavior of the client connections used by the connector. These settings allow you to manage connection limits and timeouts, ensuring efficient and reliable communication. | + + +The `actions` parameter supports the following options. + +| Field | Data type | Description | +|:------------------------|:------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `action_type` | String | Required. Sets the ML Commons API operation to use upon connection. As of OpenSearch 2.9, only `predict` is supported. | +| `method` | String | Required. Defines the HTTP method for the API call. Supports `POST` and `GET`. | +| `url` | String | Required. Sets the connection endpoint at which the action occurs. This must match the regex expression for the connection used when [adding trusted endpoints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/index#adding-trusted-endpoints). | +| `headers` | JSON object | Sets the headers used inside the request or response body. Default is `ContentType: application/json`. If your third-party ML tool requires access control, define the required `credential` parameters in the `headers` parameter. | +| `request_body` | String | Required. Sets the parameters contained in the request body of the action. The parameters must include `\"inputText\`, which specifies how users of the connector should construct the request payload for the `action_type`. | +| `pre_process_function` | String | Optional. A built-in or custom Painless script used to preprocess the input data. OpenSearch provides the following built-in preprocess functions that you can call directly:
- `connector.pre_process.cohere.embedding` for [Cohere](https://cohere.com/) embedding models
- `connector.pre_process.openai.embedding` for [OpenAI](https://openai.com/) embedding models
- `connector.pre_process.default.embedding`, which you can use to preprocess documents in neural search requests so that they are in the format that ML Commons can process with the default preprocessor (OpenSearch 2.11 or later). For more information, see [Built-in functions](#built-in-pre--and-post-processing-functions). | +| `post_process_function` | String | Optional. A built-in or custom Painless script used to post-process the model output data. OpenSearch provides the following built-in post-process functions that you can call directly:
- `connector.pre_process.cohere.embedding` for [Cohere text embedding models](https://docs.cohere.com/reference/embed)
- `connector.pre_process.openai.embedding` for [OpenAI text embedding models](https://platform.openai.com/docs/api-reference/embeddings)
- `connector.post_process.default.embedding`, which you can use to post-process documents in the model response so that they are in the format that neural search expects (OpenSearch 2.11 or later). For more information, see [Built-in functions](#built-in-pre--and-post-processing-functions). | + + +The `client_config` parameter supports the following options. + +| Field | Data type | Description | +|:---------------------|:----------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `max_connection` | Integer | The maximum number of concurrent connections that the client can establish with the server. | +| `connection_timeout` | Integer | The maximum amount of time (in seconds) that the client will wait while trying to establish a connection to the server. A timeout prevents the client from waiting indefinitely and allows it to recover from unreachable network endpoints. | +| `read_timeout` | Integer | The maximum amount of time (in seconds) that the client will wait for a response from the server after sending a request. Useful when the server is slow to respond or encounters issues while processing a request. | ## Built-in pre- and post-processing functions diff --git a/_ml-commons-plugin/remote-models/index.md b/_ml-commons-plugin/remote-models/index.md index 0b9c6d03ed..657d7254be 100644 --- a/_ml-commons-plugin/remote-models/index.md +++ b/_ml-commons-plugin/remote-models/index.md @@ -205,7 +205,18 @@ Take note of the returned `model_id` because you’ll need it to deploy the mode ## Step 4: Deploy the model -To deploy the registered model, provide its model ID from step 3 in the following request: +Starting with OpenSearch version 2.13, externally hosted models are deployed automatically by default when you send a Predict API request for the first time. To disable automatic deployment for an externally hosted model, set `plugins.ml_commons.model_auto_deploy.enable` to `false`: +```json +PUT _cluster/settings +{ + "persistent": { + "plugins.ml_commons.model_auto_deploy.enable" : "false" + } +} +``` +{% include copy-curl.html %} + +To undeploy the model, use the [Undeploy API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/undeploy-model/). ```bash POST /_plugins/_ml/models/cleMb4kBJ1eYAeTMFFg4/_deploy diff --git a/_query-dsl/minimum-should-match.md b/_query-dsl/minimum-should-match.md index 9ec65431b1..e2032b8911 100644 --- a/_query-dsl/minimum-should-match.md +++ b/_query-dsl/minimum-should-match.md @@ -26,7 +26,7 @@ GET /shakespeare/_search } ``` -In this example, the query has three optional clauses that are combined with an `OR`, so the document must match either `prince`, `king`, or `star`. +In this example, the query has three optional clauses that are combined with an `OR`, so the document must match either `prince` and `king`, or `prince` and `star`, or `king` and `star`. ## Valid values @@ -448,4 +448,4 @@ The results contain only four documents that match at least one of the optional ] } } -``` \ No newline at end of file +``` diff --git a/_search-plugins/caching/index.md b/_search-plugins/caching/index.md new file mode 100644 index 0000000000..4d0173fdc7 --- /dev/null +++ b/_search-plugins/caching/index.md @@ -0,0 +1,32 @@ +--- +layout: default +title: Caching +parent: Improving search performance +has_children: true +nav_order: 100 +--- + +# Caching + +OpenSearch relies heavily on different on-heap cache types to accelerate data retrieval, providing significant improvement in search latencies. However, cache size is limited by the amount of memory available on a node. If you are processing a larger dataset that can potentially be cached, the cache size limit causes a lot of cache evictions and misses. The increasing number of evictions impacts performance because OpenSearch needs to process the query again, causing high resource consumption. + +Prior to version 2.13, OpenSearch supported the following on-heap cache types: + +- **Request cache**: Caches the local results on each shard. This allows frequently used (and potentially resource-heavy) search requests to return results almost instantly. +- **Query cache**: The shard-level query cache caches common data from similar queries. The query cache is more granular than the request cache and can cache data that is reused in different queries. +- **Field data cache**: The field data cache contains field data and global ordinals, which are both used to support aggregations on certain field types. + +## Additional cache stores +**Introduced 2.13** +{: .label .label-purple } + +This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/10024). +{: .warning} + +In addition to existing OpenSearch custom on-heap cache stores, cache plugins provide the following cache stores: + +- **Disk cache**: This cache stores the precomputed result of a query on disk. You can use a disk cache to cache much larger datasets, provided that the disk latencies are acceptable. +- **Tiered cache**: This is a multi-level cache, in which each tier has its own characteristics and performance levels. For example, a tiered cache can contain on-heap and disk tiers. By combining different tiers, you can achieve a balance between cache performance and size. To learn more, see [Tiered cache]({{site.url}}{{site.baseurl}}/search-plugins/caching/tiered-cache/). + +In OpenSearch 2.13, the request cache is integrated with cache plugins. You can use a tiered or disk cache as a request-level cache. +{: .note} \ No newline at end of file diff --git a/_search-plugins/caching/tiered-cache.md b/_search-plugins/caching/tiered-cache.md new file mode 100644 index 0000000000..3842ebe5a9 --- /dev/null +++ b/_search-plugins/caching/tiered-cache.md @@ -0,0 +1,82 @@ +--- +layout: default +title: Tiered cache +parent: Caching +grand_parent: Improving search performance +nav_order: 10 +--- + +# Tiered cache + +This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/10024). +{: .warning} + +A tiered cache is a multi-level cache, in which each tier has its own characteristics and performance levels. By combining different tiers, you can achieve a balance between cache performance and size. + +## Types of tiered caches + +OpenSearch 2.13 provides an implementation of _tiered spillover cache_. This implementation spills the evicted items from upper to lower tiers. The upper tier is smaller in size but offers better latency, like the on-heap tier. The lower tier is larger in size but is slower in terms of latency compared to the upper tier. A disk cache is an example of a lower tier. OpenSearch 2.13 offers on-heap and disk tiers. + +## Enabling a tiered cache + +To enable a tiered cache, configure the following setting: + +```yaml +opensearch.experimental.feature.pluggable.caching.enabled: true +``` +{% include copy.html %} + +For more information about ways to enable experimental features, see [Experimental feature flags]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/). + +## Installing required plugins + +A tiered cache provides a way to plug in any disk or on-heap tier implementation. You can install the plugins you intend to use in the tiered cache. As of OpenSearch 2.13, the available cache plugin is the `cache-ehcache` plugin. This plugin provides a disk cache implementation to use within a tiered cache as a disk tier. + +A tiered cache will fail to initialize if the `cache-ehcache` plugin is not installed or disk cache properties are not set. +{: .warning} + +## Tiered cache settings + +In OpenSearch 2.13, a request cache can use a tiered cache. To begin, configure the following settings in the `opensearch.yml` file. + +### Cache store name + +Set the cache store name to `tiered_spillover` to use the OpenSearch-provided tiered spillover cache implementation: + +```yaml +indices.request.cache.store.name: tiered_spillover: true +``` +{% include copy.html %} + +### Setting on-heap and disk store tiers + +The `opensearch_onheap` setting is the built-in on-heap cache available in OpenSearch. The `ehcache_disk` setting is the disk cache implementation from [Ehcache](https://www.ehcache.org/). This requires installing the `cache-ehcache` plugin: + +```yaml +indices.request.cache.tiered_spillover.onheap.store.name: opensearch_onheap +indices.request.cache.tiered_spillover.disk.store.name: ehcache_disk +``` +{% include copy.html %} + +For more information about installing non-bundled plugins, see [Additional plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/#additional-plugins). + +### Configuring on-heap and disk stores + +The following table lists the cache store settings for the `opensearch_onheap` store. + +Setting | Default | Description +:--- | :--- | :--- +`indices.request.cache.opensearch_onheap.size` | 1% of the heap | The size of the on-heap cache. Optional. +`indices.request.cache.opensearch_onheap.expire` | `MAX_VALUE` (disabled) | Specify a time-to-live (TTL) for the cached results. Optional. + +The following table lists the disk cache store settings for the `ehcache_disk` store. + +Setting | Default | Description +:--- | :--- | :--- +`indices.request.cache.ehcache_disk.max_size_in_bytes` | `1073741824` (1 GB) | Defines the size of the disk cache. Optional. +`indices.request.cache.ehcache_disk.storage.path` | `""` | Defines the storage path for the disk cache. Required. +`indices.request.cache.ehcache_disk.expire_after_access` | `MAX_VALUE` (disabled) | Specify a time-to-live (TTL) for the cached results. Optional. +`indices.request.cache.ehcache_disk.alias` | `ehcacheDiskCache#INDICES_REQUEST_CACHE` (this is an example of request cache) | Specify an alias for the disk cache. Optional. +`indices.request.cache.ehcache_disk.segments` | `16` | Defines the number of segments the disk cache is separated into. Used for concurrency. Optional. +`indices.request.cache.ehcache_disk.concurrency` | `1` | Defines the number of distinct write queues created for the disk store, where a group of segments share a write queue. Optional. + diff --git a/_search-plugins/concurrent-segment-search.md b/_search-plugins/concurrent-segment-search.md index 58b8d9a8ce..0bb7657937 100644 --- a/_search-plugins/concurrent-segment-search.md +++ b/_search-plugins/concurrent-segment-search.md @@ -27,7 +27,7 @@ By default, concurrent segment search is disabled on the cluster. You can enable - Cluster level - Index level -The index-level setting takes priority over the cluster-level setting. Thus, if the cluster setting is enabled but the index setting is disabled, then concurrent segment search will be disabled for that index. +The index-level setting takes priority over the cluster-level setting. Thus, if the cluster setting is enabled but the index setting is disabled, then concurrent segment search will be disabled for that index. Because of this, the index-level setting is not evaluated unless it is explicitly set, regardless of the default value configured for the setting. You can retrieve the current value of the index-level setting by calling the [Index Settings API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/get-settings/) and omitting the `?include_defaults` query parameter. {: .note} To enable concurrent segment search for all indexes in the cluster, set the following dynamic cluster setting: diff --git a/_search-plugins/hybrid-search.md b/_search-plugins/hybrid-search.md index ebd014b0de..b0fb4d5bef 100644 --- a/_search-plugins/hybrid-search.md +++ b/_search-plugins/hybrid-search.md @@ -146,7 +146,9 @@ PUT /_search/pipeline/nlp-search-pipeline To perform hybrid search on your index, use the [`hybrid` query]({{site.url}}{{site.baseurl}}/query-dsl/compound/hybrid/), which combines the results of keyword and semantic search. -The following example request combines two query clauses---a neural query and a `match` query. It specifies the search pipeline created in the previous step as a query parameter: +#### Example: Combining a neural query and a match query + +The following example request combines two query clauses---a `neural` query and a `match` query. It specifies the search pipeline created in the previous step as a query parameter: ```json GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline @@ -161,7 +163,7 @@ GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline "queries": [ { "match": { - "text": { + "passage_text": { "query": "Hi world" } } @@ -216,3 +218,355 @@ The response contains the matching document: } } ``` +{% include copy-curl.html %} + +#### Example: Combining a match query and a term query + +The following example request combines two query clauses---a `match` query and a `term` query. It specifies the search pipeline created in the previous step as a query parameter: + +```json +GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline +{ + "_source": { + "exclude": [ + "passage_embedding" + ] + }, + "query": { + "hybrid": { + "queries": [ + { + "match":{ + "passage_text": "hello" + } + }, + { + "term":{ + "passage_text":{ + "value":"planet" + } + } + } + ] + } + } +} +``` +{% include copy-curl.html %} + +The response contains the matching documents: + +```json +{ + "took": 11, + "timed_out": false, + "_shards": { + "total": 2, + "successful": 2, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 0.7, + "hits": [ + { + "_index": "my-nlp-index", + "_id": "2", + "_score": 0.7, + "_source": { + "id": "s2", + "passage_text": "Hi planet" + } + }, + { + "_index": "my-nlp-index", + "_id": "1", + "_score": 0.3, + "_source": { + "id": "s1", + "passage_text": "Hello world" + } + } + ] + } +} +``` +{% include copy-curl.html %} + +## Hybrid search with post-filtering +**Introduced 2.13** +{: .label .label-purple } + +You can perform post-filtering on hybrid search results by providing the `post_filter` parameter in your query. + +The `post_filter` clause is applied after the search results have been retrieved. Post-filtering is useful for applying additional filters to the search results without impacting the scoring or the order of the results. + +Post-filtering does not impact document relevance scores or aggregation results. +{: .note} + +#### Example: Post-filtering + +The following example request combines two query clauses---a `term` query and a `match` query. This is the same query as in the [preceding example](#example-combining-a-match-query-and-a-term-query), but it contains a `post_filter`: + +```json +GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline +{ + "query": { + "hybrid":{ + "queries":[ + { + "match":{ + "passage_text": "hello" + } + }, + { + "term":{ + "passage_text":{ + "value":"planet" + } + } + } + ] + } + + }, + "post_filter":{ + "match": { "passage_text": "world" } + } +} + +``` +{% include copy-curl.html %} + +Compare the results to the results without post-filtering in the [preceding example](#example-combining-a-match-query-and-a-term-query). Unlike the preceding example response, which contains two documents, the response in this example contains one document because the second document is filtered using post-filtering: + +```json +{ + "took": 18, + "timed_out": false, + "_shards": { + "total": 2, + "successful": 2, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 0.3, + "hits": [ + { + "_index": "my-nlp-index", + "_id": "1", + "_score": 0.3, + "_source": { + "id": "s1", + "passage_text": "Hello world" + } + } + ] + } +} +``` + + +## Combining hybrid search and aggregations +**Introduced 2.13** +{: .label .label-purple } + +You can enhance search results by combining a hybrid query clause with any aggregation that OpenSearch supports. Aggregations allow you to use OpenSearch as an analytics engine. For more information about aggregations, see [Aggregations]({{site.url}}{{site.baseurl}}/aggregations/). + +Most aggregations are performed on the subset of documents that is returned by a hybrid query. The only aggregation that operates on all documents is the [`global`]({{site.url}}{{site.baseurl}}/aggregations/bucket/global/) aggregation. + +To use aggregations with a hybrid query, first create an index. Aggregations are typically used on fields of special types, like `keyword` or `integer`. The following example creates an index with several such fields: + +```json +PUT /my-nlp-index +{ + "settings": { + "number_of_shards": 2 + }, + "mappings": { + "properties": { + "doc_index": { + "type": "integer" + }, + "doc_keyword": { + "type": "keyword" + }, + "category": { + "type": "keyword" + } + } + } +} +``` +{% include copy-curl.html %} + +The following request ingests six documents into your new index: + +```json +POST /_bulk +{ "index": { "_index": "my-nlp-index" } } +{ "category": "permission", "doc_keyword": "workable", "doc_index": 4976, "doc_price": 100} +{ "index": { "_index": "my-nlp-index" } } +{ "category": "sister", "doc_keyword": "angry", "doc_index": 2231, "doc_price": 200 } +{ "index": { "_index": "my-nlp-index" } } +{ "category": "hair", "doc_keyword": "likeable", "doc_price": 25 } +{ "index": { "_index": "my-nlp-index" } } +{ "category": "editor", "doc_index": 9871, "doc_price": 30 } +{ "index": { "_index": "my-nlp-index" } } +{ "category": "statement", "doc_keyword": "entire", "doc_index": 8242, "doc_price": 350 } +{ "index": { "_index": "my-nlp-index" } } +{ "category": "statement", "doc_keyword": "idea", "doc_index": 5212, "doc_price": 200 } +{ "index": { "_index": "index-test" } } +{ "category": "editor", "doc_keyword": "bubble", "doc_index": 1298, "doc_price": 130 } +{ "index": { "_index": "index-test" } } +{ "category": "editor", "doc_keyword": "bubble", "doc_index": 521, "doc_price": 75 } +``` +{% include copy-curl.html %} + +Now you can combine a hybrid query clause with a `min` aggregation: + +```json +GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline +{ + "query": { + "hybrid": { + "queries": [ + { + "term": { + "category": "permission" + } + }, + { + "bool": { + "should": [ + { + "term": { + "category": "editor" + } + }, + { + "term": { + "category": "statement" + } + } + ] + } + } + ] + } + }, + "aggs": { + "total_price": { + "sum": { + "field": "doc_price" + } + }, + "keywords": { + "terms": { + "field": "doc_keyword", + "size": 10 + } + } + } +} +``` +{% include copy-curl.html %} + +The response contains the matching documents and the aggregation results: + +```json +{ + "took": 9, + "timed_out": false, + "_shards": { + "total": 2, + "successful": 2, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 4, + "relation": "eq" + }, + "max_score": 0.5, + "hits": [ + { + "_index": "my-nlp-index", + "_id": "mHRPNY4BlN82W_Ar9UMY", + "_score": 0.5, + "_source": { + "doc_price": 100, + "doc_index": 4976, + "doc_keyword": "workable", + "category": "permission" + } + }, + { + "_index": "my-nlp-index", + "_id": "m3RPNY4BlN82W_Ar9UMY", + "_score": 0.5, + "_source": { + "doc_price": 30, + "doc_index": 9871, + "category": "editor" + } + }, + { + "_index": "my-nlp-index", + "_id": "nXRPNY4BlN82W_Ar9UMY", + "_score": 0.5, + "_source": { + "doc_price": 200, + "doc_index": 5212, + "doc_keyword": "idea", + "category": "statement" + } + }, + { + "_index": "my-nlp-index", + "_id": "nHRPNY4BlN82W_Ar9UMY", + "_score": 0.5, + "_source": { + "doc_price": 350, + "doc_index": 8242, + "doc_keyword": "entire", + "category": "statement" + } + } + ] + }, + "aggregations": { + "total_price": { + "value": 680 + }, + "doc_keywords": { + "doc_count_error_upper_bound": 0, + "sum_other_doc_count": 0, + "buckets": [ + { + "key": "entire", + "doc_count": 1 + }, + { + "key": "idea", + "doc_count": 1 + }, + { + "key": "workable", + "doc_count": 1 + } + ] + } + } +} +``` \ No newline at end of file diff --git a/_search-plugins/knn/approximate-knn.md b/_search-plugins/knn/approximate-knn.md index 74cf7e39f5..16d1a7e686 100644 --- a/_search-plugins/knn/approximate-knn.md +++ b/_search-plugins/knn/approximate-knn.md @@ -287,9 +287,15 @@ Not every method supports each of these spaces. Be sure to check out [the method
Lucene:\[ score = {2 - d \over 2}\]
Lucene: + \[ d(\mathbf{x}, \mathbf{y}) = {\mathbf{x} · \mathbf{y}} = \sum_{i=1}^n x_i y_i \] +
Lucene: + \[ \text{If} d > 0, score = d + 1 \] \[\text{If} d \le 0\] \[score = {1 \over 1 + (-1 · d) }\] +