Show only intersecting buckets to the Adjacency matrix aggregation #10226

brusic · 2023-09-25T23:10:49Z

EDIT: Rebasing the code to add the signoff created an inconsistency with the branch. Never used Signed-off-by, willing to start a new branch since this one appears to have prior commits.

Description

Show only intersecting buckets to the Adjacency matrix aggregation

The Adjacency matrix aggregation will compute and return all buckets related to any of the combinations of the filters specified, including hits on a single filter. However, sometimes a user would want to return only the buckets related to an intersection and not for a single filter. An optional show_only_intersecting parameter would stop buckets resulting from a single filter being hit to be created.

Related Issues

#8832

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

… for ingestion (opensearch-project#9190) --------- Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…-project#9082) This is the first round refactor of the foundation Task classes from the server module to core and common library. This is split into separate commits due to control the surface area impacted by the change. This commit refactors the following classes: * o.o.action.ActionFuture -> :libs:opensearch-common * o.o.action.ActionListener -> :libs:opensearch-core * o.o.action.ActionResponse -> :libs:opensearch-core * o.o.action.NotifyOnceListener -> :libs:opensearch-core * o.o.common.compress.Compressor -> :libs:opensearch-core * o.o.node.ReportingService -> o.o.core.service in :libs:opensearch-core * o.o.tasks.TaskCancelledExceptoin -> :libs:opensearch-core * o.o.tasks.TaskId -> :libs:opensearch-core * o.o.common.xcontent.ObjectParserHelper -> :libs:opensearch-core The following task resource tracking classes are refactored into a new resourcetracker namespace under the o.o.core.tasks packed in :libs:opensearch-core * o.o.tasks.ResourceStats * o.o.tasks.ResourceStatsType * o.o.tasks.ResourceUsageMetric * o.o.tasks.TaskResourceStats * o.o.tasks.TaskResourceUsage * o.o.tasks.TaskThreadUsage * o.o.tasks.ThreadResourceInfo These refactors are part of the effort to split foundation classes into core libraries in support of modularity for serverless and cloud native implementations. Signed-off-by: Nicholas Walter Knize <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…nsearch-project#9221) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…/repository-azure (opensearch-project#9231) * Bump com.azure:azure-storage-blob in /plugins/repository-azure Bumps [com.azure:azure-storage-blob](https://github.com/Azure/azure-sdk-for-java) from 12.22.3 to 12.23.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.22.3...azure-storage-blob_12.23.0) --- updated-dependencies: - dependency-name: com.azure:azure-storage-blob dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Ivan Brusic <[email protected]>

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <[email protected]> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <[email protected]> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <[email protected]> * Add more shard level tests. Signed-off-by: Marc Handalian <[email protected]> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <[email protected]> * Self review. Signed-off-by: Marc Handalian <[email protected]> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…9227) * Bump com.diffplug.spotless from 6.19.0 to 6.20.0 Bumps com.diffplug.spotless from 6.19.0 to 6.20.0. --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Ivan Brusic <[email protected]>

… to 1.43.3 (opensearch-project#10126) Signed-off-by: Ivan Brusic <[email protected]>

…ore (opensearch-project#10135) * Revert "Optimize read write lock constructs during translog upload to remote store (opensearch-project#9636)" This reverts commit b5cc002. Signed-off-by: Ashish Singh <[email protected]> * Downgrade writelock to readlock during translog upload Signed-off-by: Ashish Singh <[email protected]> --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

Provide service accounts tokens to extensions This change adds a new transport action which passes the extension a string representation of its service account auth token. This token is created by the TokenManager interface implementation. The token is expected to be an encoded basic auth credential string which can be used by the extension to interact with its own system index. Signed-off-by: Stephen Crawford <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> Signed-off-by: Peter Nied <[email protected]> Co-authored-by: Owais Kazi <[email protected]> Co-authored-by: Peter Nied <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…t#10178) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

* Add support for encrypted async blob read Signed-off-by: Kunal Kotwani <[email protected]> * Add async blob read support for encrypted containers Signed-off-by: Kunal Kotwani <[email protected]> --------- Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…-project#9710) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…t#10036) Signed-off-by: Ivan Brusic <[email protected]>

…/fixtures/hdfs-fixture (opensearch-project#10206) * Bump org.xerial.snappy:snappy-java in /test/fixtures/hdfs-fixture Bumps [org.xerial.snappy:snappy-java](https://github.com/xerial/snappy-java) from 1.1.10.3 to 1.1.10.4. - [Release notes](https://github.com/xerial/snappy-java/releases) - [Commits](xerial/snappy-java@v1.1.10.3...v1.1.10.4) --- updated-dependencies: - dependency-name: org.xerial.snappy:snappy-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Ivan Brusic <[email protected]>

…ectly (opensearch-project#10089) * Fix remove ingest processor handing ignore_missing parameter not correctly Signed-off-by: Gao Binlong <[email protected]> * modify change log Signed-off-by: Gao Binlong <[email protected]> * Fix test failure and optimize some code Signed-off-by: Gao Binlong <[email protected]> * Minor change Signed-off-by: Gao Binlong <[email protected]> * Modify test code Signed-off-by: Gao Binlong <[email protected]> * Split a test method to two Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

* Bump codecov/codecov-action from 2 to 3 Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2 to 3. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v2...v3) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Ivan Brusic <[email protected]>

* [Tracing Framework] Add support for SpanKind Signed-off-by: Gagan Juneja <[email protected]> * Add CHANGELOG Signed-off-by: Gagan Juneja <[email protected]> * Fix test case Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> --------- Signed-off-by: Gagan Juneja <[email protected]> Co-authored-by: Gagan Juneja <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…25.0 in /plugins/repository-gcs (opensearch-project#10208) * Bump com.google.api.grpc:proto-google-common-protos Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) from 2.10.0 to 2.25.0. - [Release notes](https://github.com/googleapis/sdk-platform-java/releases) - [Changelog](https://github.com/googleapis/sdk-platform-java/blob/main/CHANGELOG.md) - [Commits](googleapis/sdk-platform-java@v2.10.0...v2.25.0) --- updated-dependencies: - dependency-name: com.google.api.grpc:proto-google-common-protos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Andriy Redko <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

Signed-off-by: Andrey Pleskach <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…rsecting

github-actions · 2023-09-25T23:27:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26289/
CommitID: 3d66a5f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-25T23:33:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26290/
CommitID: 8fc0cb5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-25T23:46:45Z

Compatibility status:

Checks if related components are compatible with change 188ff84

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-26T00:03:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26288/
CommitID: 0c9a4ca
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-26T00:18:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26292/
CommitID: 188ff84
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-10-26T15:20:05Z

This PR is stalled because it has been open for 30 days with no activity.

brusic · 2024-01-03T23:08:12Z

transitioned to #11733

gbbafna and others added 6 commits September 25, 2023 16:12

[Remote Store] Fix couple of Remote Store flaky test and use bulk api…

c2c20d7

… for ingestion (opensearch-project#9190) --------- Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

Automatically update GitHub Actions dependencies with dependabot (ope…

a995c2b

…nsearch-project#9221) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

Poojita-Raj and others added 13 commits September 25, 2023 16:15

Bump com.google.http-client:google-http-client-jackson2 from 1.43.2…

2228e76

… to 1.43.3 (opensearch-project#10126) Signed-off-by: Ivan Brusic <[email protected]>

Reenable multipart repository within multipart ITs (opensearch-projec…

cb7142c

…t#10178) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

Add async segment file download support from remote store (opensearch…

74aad97

…-project#9710) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

Honor max segment size during only_expunge_deletes (opensearch-projec…

5c0e568

…t#10036) Signed-off-by: Ivan Brusic <[email protected]>

Bump BouncyCastle to 1.76 (opensearch-project#10219)

3d66a5f

Signed-off-by: Andrey Pleskach <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

brusic force-pushed the brusic/8832-show-only-intersecting branch from 0c9a4ca to 3d66a5f Compare September 25, 2023 23:15

Merge remote-tracking branch 'origin' into brusic/8832-show-only-inte…

8fc0cb5

…rsecting

brusic marked this pull request as draft September 25, 2023 23:18

brusic changed the title ~~Brusic/8832 show only intersecting~~ Show only intersecting buckets to the Adjacency matrix aggregation Sep 25, 2023

brusic marked this pull request as ready for review September 25, 2023 23:24

Fix errant keystroke

188ff84

opensearch-trigger-bot bot added the stalled Issues that have stalled label Oct 26, 2023

brusic mentioned this pull request Jan 3, 2024

Show only intersecting buckets to the Adjacency matrix aggregation #11733

Merged

6 tasks

brusic closed this Jan 3, 2024

brusic deleted the brusic/8832-show-only-intersecting branch January 3, 2024 23:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Show only intersecting buckets to the Adjacency matrix aggregation #10226

Show only intersecting buckets to the Adjacency matrix aggregation #10226

brusic commented Sep 25, 2023 •

edited

Loading

github-actions bot commented Sep 25, 2023

github-actions bot commented Sep 25, 2023

github-actions bot commented Sep 25, 2023 •

edited

Loading

github-actions bot commented Sep 26, 2023

github-actions bot commented Sep 26, 2023

opensearch-trigger-bot bot commented Oct 26, 2023

brusic commented Jan 3, 2024

Show only intersecting buckets to the Adjacency matrix aggregation #10226

Show only intersecting buckets to the Adjacency matrix aggregation #10226

Conversation

brusic commented Sep 25, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Sep 25, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 25, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 25, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 26, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Oct 26, 2023

brusic commented Jan 3, 2024

brusic commented Sep 25, 2023 •

edited

Loading

github-actions bot commented Sep 25, 2023 •

edited

Loading