Skip to content

feat(search): geohash aggregation - #3272

Draft
dschmidt wants to merge 10 commits into
feat/search-bleve-geohash-siblingfrom
feat/search-geohash-aggregation
Draft

dschmidt wants to merge 10 commits into
feat/search-bleve-geohash-siblingfrom
feat/search-geohash-aggregation

Conversation

@dschmidt

@dschmidt dschmidt commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

Description

AggregationOption.geohash_precision (1-12) turns an aggregation on a geopoint field into a geohash grid: bucket key is the cell, value the doc count. Rejected on non-geo fields and precisions beyond 12.

  • OpenSearch: native geohash_grid on the geopoint sibling.
  • bleve: terms facet on the geohash sibling from feat(search): geohash support in bleve #3519, restricted to the terms of the requested precision; nested, folded from doc values like the other levels.

Stacked on #3211 and #3519. No mapping change.

How Has This Been Tested

  • AGG-17 cells at precision 5, AGG-18 MimeType nested in cells at precision 3, AGG-19 non-geo field, AGG-20 precision 13, on both engines

Types of changes

  • New feature (non-breaking change which adds functionality)

@codacy-production

codacy-production Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 407 complexity

Metric Results
Complexity 407

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@dschmidt
dschmidt force-pushed the feat/graph-search-query branch from 1b61ef6 to fb1e41b Compare August 18, 2026 16:51
@dschmidt
dschmidt force-pushed the feat/search-geohash-aggregation branch from 59aacc8 to c775513 Compare August 18, 2026 17:00
@dschmidt
dschmidt force-pushed the feat/graph-search-query branch 9 times, most recently from 2513263 to 71ca404 Compare September 3, 2026 08:33
@dschmidt
dschmidt force-pushed the feat/search-geohash-aggregation branch from c775513 to 2d46411 Compare September 7, 2026 22:14
dschmidt added a commit that referenced this pull request Sep 8, 2026
The same tokenizer with tag_depth turns a geohash into depth-tagged prefixes, so a terms facet with TermPrefix is a geohash grid at that precision. One location_geohash field per geopoint, part of the v5 schema so #3272 needs no further bump. OpenSearch maps it unindexed, geohash_grid works on the geo_point.
dschmidt added a commit that referenced this pull request Sep 10, 2026
Path was a keyword, so the descendant lookup behind delete/move/restore/purge, the scoped search and the KQL path predicate expanded into one term searcher per descendant and OOM-killed the server on large folders (#1269, #3469).

Path is now analyzed into its ancestor prefixes, like path_hierarchy in OpenSearch: ./a/b.txt becomes ., ./a, ./a/b.txt. A folder's descendants are every document carrying the folder's path as a term, so all three call sites are a single term query. Schema 4 -> 5, v4 never shipped.

The same tokenizer with tag_depth is registered as the geohash analyzer, so #3272 can add its geohash field without another schema change.
dschmidt added a commit that referenced this pull request Sep 10, 2026
Path was a keyword, so the descendant lookup behind delete/move/restore/purge, the scoped search and the KQL path predicate expanded into one term searcher per descendant and OOM-killed the server on large folders (#1269, #3469).

Path is now analyzed into its ancestor prefixes, like path_hierarchy in OpenSearch: ./a/b.txt becomes ., ./a, ./a/b.txt. A folder's descendants are every document carrying the folder's path as a term, so all three call sites are a single term query. Schema 4 -> 5, v4 never shipped.

The same tokenizer with tag_depth is registered as the geohash analyzer, so #3272 can add its geohash field without another schema change.
dschmidt added a commit that referenced this pull request Sep 10, 2026
Path was a keyword, so the descendant lookup behind delete/move/restore/purge, the scoped search and the KQL path predicate expanded into one term searcher per descendant and OOM-killed the server on large folders (#1269, #3469).

Path is now analyzed into its ancestor prefixes, like path_hierarchy in OpenSearch: ./a/b.txt becomes ., ./a, ./a/b.txt. A folder's descendants are every document carrying the folder's path as a term, so all three call sites are a single term query. Schema 4 -> 5, v4 never shipped.

The same tokenizer with tag_depth is registered as the geohash analyzer, so #3272 can add its geohash field without another schema change.
dschmidt added a commit that referenced this pull request Sep 10, 2026
Path was a keyword, so the descendant lookup behind delete/move/restore/purge, the scoped search and the KQL path predicate expanded into one term searcher per descendant and OOM-killed the server on large folders (#1269, #3469).

Path is now analyzed into its ancestor prefixes, like path_hierarchy in OpenSearch: ./a/b.txt becomes ., ./a, ./a/b.txt. A folder's descendants are every document carrying the folder's path as a term, so all three call sites are a single term query. Schema 4 -> 5, v4 never shipped.

The same tokenizer with tag_depth is registered as the geohash analyzer, so #3272 can add its geohash field without another schema change.
dschmidt added a commit that referenced this pull request Sep 10, 2026
Path was a keyword, so the descendant lookup behind delete/move/restore/purge, the scoped search and the KQL path predicate expanded into one term searcher per descendant and OOM-killed the server on large folders (#1269, #3469).

Path is now analyzed into its ancestor prefixes, like path_hierarchy in OpenSearch: ./a/b.txt becomes ., ./a, ./a/b.txt. A folder's descendants are every document carrying the folder's path as a term, so all three call sites are a single term query. Schema 4 -> 5, v4 never shipped.

The same tokenizer with tag_depth is registered as the geohash analyzer, so #3272 can add its geohash field without another schema change.
dschmidt added a commit that referenced this pull request Sep 11, 2026
OpenSearch buckets a geohash_grid aggregation on the geo_point itself, bleve has no such aggregation. bleve now maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into depth-tagged prefixes, so a terms facet with TermPrefix "<precision>/" is the grid at that precision (#3272). Only bleve knows the field, the shared document shape and OpenSearch are untouched; the mapping change is additive.
dschmidt added a commit that referenced this pull request Sep 11, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into depth-tagged prefixes, so a terms facet with TermPrefix "<precision>/" is a geohash grid (#3272). bleve only, additive on the v5 schema.
dschmidt added a commit that referenced this pull request Sep 11, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into depth-tagged prefixes, so a terms facet with TermPrefix "<precision>/" is a geohash grid (#3272). bleve only, additive on the v5 schema.
dschmidt added a commit that referenced this pull request Sep 11, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
dschmidt added a commit that referenced this pull request Sep 11, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
dschmidt added a commit that referenced this pull request Sep 11, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
dschmidt added a commit that referenced this pull request Sep 11, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
@dschmidt
dschmidt force-pushed the feat/graph-search-query branch from fef1116 to 7f7ecbf Compare September 11, 2026 14:30
dschmidt added a commit that referenced this pull request Sep 11, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
@dschmidt
dschmidt force-pushed the feat/search-geohash-aggregation branch from 2d46411 to d7bdce8 Compare September 11, 2026 14:33
@dschmidt dschmidt changed the title feat(search): geohash aggregation (bleve + opensearch) feat(search): geohash aggregation Sep 11, 2026
@dschmidt
dschmidt changed the base branch from feat/graph-search-query to feat/search-bleve-geohash-sibling September 11, 2026 14:33
@dschmidt
dschmidt added this pull request to stack #3520 September 11, 2026 14:34
Generated from opencloud-eu/libre-graph-api#34 rebased onto main: POST /search/query with hits, aggregations and metrics.
MS-Graph-style search query endpoint: hits from all accessible spaces, with from/size paging, remote items for hits from shared spaces, thumbnails on $expand, and the effective permission actions on every hit. The search proto gains the aggregation types of the graph API and the endpoint passes aggregations through to the search service, which does not evaluate them yet.
dschmidt added a commit that referenced this pull request Sep 12, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
@dschmidt
dschmidt force-pushed the feat/search-bleve-geohash-sibling branch from f017713 to 9b4883b Compare September 12, 2026 15:34
@dschmidt
dschmidt force-pushed the feat/search-geohash-aggregation branch from d7bdce8 to c65b323 Compare September 12, 2026 15:34
dschmidt added a commit that referenced this pull request Sep 12, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
@dschmidt
dschmidt force-pushed the feat/search-bleve-geohash-sibling branch from 9b4883b to 3d449df Compare September 12, 2026 15:38
@dschmidt
dschmidt force-pushed the feat/search-geohash-aggregation branch from c65b323 to ffbdf81 Compare September 12, 2026 15:38
dschmidt added a commit that referenced this pull request Sep 12, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
@dschmidt
dschmidt force-pushed the feat/search-bleve-geohash-sibling branch from 3d449df to f0f9fce Compare September 12, 2026 16:23
@dschmidt
dschmidt force-pushed the feat/search-geohash-aggregation branch from ffbdf81 to db0b7b1 Compare September 12, 2026 16:23
Term buckets on both engines: bleve facets, OpenSearch terms aggregations, buckets merged across spaces and post-processed per BucketDefinition (minimum count, sort, size). Terms aggregations on numeric fields are rejected at the graph endpoint. Pinned in the parity suite as AGG-01 to AGG-03. Ranges, metrics and sub-aggregations are declined until the engines evaluate them.
Numeric and date ranges on both engines: bleve numeric and date-time range facets, OpenSearch range and date_range aggregations, a bound that is neither number nor date is rejected. A terms aggregation on a numeric field now points to ranges as the alternative. AGG-04, 05 and 07 to 09 in the parity suite.
Sum, min, max and avg on both engines. bleve facets cannot compute, so a collector hooked into the document-match handler folds the metric from doc values, one pass over every match, before the top-n cut. Avg travels as sum and count so the service can reduce it across spaces. AGG-06 in the parity suite.
Sub-aggregations on both engines. bleve folds child buckets from doc values through the same collector as the metrics, terms and ranges alike; the service unions nested buckets across spaces. AGG-10 to AGG-16 in the parity suite.
Every bucket carries an opaque aggregationFilterToken; a search request can pass tokens back as aggregationFilters. The service decodes them into KQL fragments once, the engines AND them into the query as exact, case-sensitive matches.
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
A geohash_precision aggregation runs as a terms facet on the geohash sibling of the geopoint, restricted to the terms tagged with the requested precision; nested, it folds the same terms from doc values. The cells match OpenSearch's geohash_grid, pinned in the parity suite.
@dschmidt
dschmidt force-pushed the feat/search-bleve-geohash-sibling branch from f0f9fce to 96dd924 Compare September 12, 2026 16:34
@dschmidt
dschmidt force-pushed the feat/search-geohash-aggregation branch from db0b7b1 to fbc1806 Compare September 12, 2026 16:34
@dschmidt
dschmidt removed this pull request from stack #3520 October 5, 2026 06:48
dschmidt added a commit that referenced this pull request Oct 6, 2026
bleve maps a <name>_geohash field next to every geopoint field and writes the geohash from the batch, analyzed into one depth-tagged term per precision. Groundwork for a geohash grid on bleve (#3272), nothing queries it yet. bleve only, additive on the v5 schema.
@dschmidt
dschmidt force-pushed the feat/search-bleve-geohash-sibling branch from 96dd924 to 58c52b7 Compare October 6, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant