Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

issue253 Use extend with overlap with retile in vectorize #360

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
+28,713 −15
Open

issue253 Use extend with overlap with retile in vectorize #360

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 24 additions & 6 deletions openeo-geotrellis/src/main/scala/org/openeo/geotrellis/OpenEOProcesses.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -617,13 +617,19 @@ class OpenEOProcesses extends Serializable {
val maxExtent = datacube.metadata.extent
//naive approach: combine tiles and hope that we don't exceed the max size
//if we exceed the max, vectorize will run on separate tiles, and we'll need to merge results
val newCols = Math.min(256*20,layout.cols)
val newRows = Math.min(256*20,layout.rows)
val newCols = Math.min(256,layout.cols)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I suggest to still work with a larger tile size by default, maybe 256*4, avoiding that we make an immediate change from very coarse tiling to small tiles.
It is always possible that certain edge cases were not detected in our testing, and by doing it like this we do a more gradual rollout.
It is however important to ensure that the unit test also continues to work on the more complex case. Making this tile size configurable would be one way to achieve that.

val newRows = Math.min(256,layout.rows)
val overlap = 8

val singleBandLayer: TileLayerRDD[K] = datacube.withContext(_.mapValues(_.band(0)))
val retiled = singleBandLayer.regrid(newCols.intValue(),newRows.intValue())
val retiled = retileGeneric(datacube,newCols.toInt,newRows.toInt,overlap,overlap)
val resolution = retiled.metadata.cellSize
val retiledSingleBand = retiled.withContext(_.mapValues(_.band(0)))
// Perform the actual vectorization.
val vectorizedValues: RDD[(K, List[PolygonFeature[Int]])] = retiled.toRasters.mapValues(_.crop(maxExtent,Crop.Options(force=true,clamp=true)).toVector())
val vectorizedValues: RDD[(K, List[PolygonFeature[Int]])] = retiledSingleBand.toRasters.mapValues(v => {
val overlapExtent = v.extent.buffer(resolution.width*overlap,resolution.height*overlap)
val overlap_cube = new Raster[Tile](v.tile,overlapExtent)
overlap_cube.crop(maxExtent,Crop.Options(force=true,clamp=true)).toVector()
})
// We don't require spatial partitioning for features, so we can group by (Time, Band) instead.
// In the meantime we construct the feature ids as they will appear in the geojson file.
val featuresWithId: RDD[(String, List[PolygonFeature[Int]])] = vectorizedValues.map(kv => {
Expand All @@ -641,9 +647,21 @@ class OpenEOProcesses extends Serializable {
def featuresToGeojson(features: Array[(String, List[PolygonFeature[Int]])], crs: CRS): Json = {
// Add index to each feature id, so final id will be 'date_band_index'.
val geojsonFeaturesWithId: Array[Json] = features.flatMap((v) => {
val polygons_per_values: List[List[PolygonFeature[Int]]] = v._2.map(_.data).distinct.map(x => v._2.filter(_.data == x))
val combined_polygons = polygons_per_values.map(
x =>{
val union = x.foldRight(Option.empty[Geometry])((a,b)=>{
if (b.isEmpty)
Option(a.geom)
else
Option(b.get.union(a))
})
new Feature(union.get, x.head.data)
}
)
val key: String = v._1 // (Time, Band) key.
// Geojson lists properties as a map.
val feats: List[PolygonFeature[Map[String,Int]]] = v._2.map(_.mapData(v => immutable.Map("value" -> v)))
val feats = combined_polygons.map(_.mapData(v => immutable.Map("value" -> v)))
feats.zipWithIndex.map({case (f,i) => f.asJson.deepMerge(Json.obj("id" -> (key + "_" + i).asJson))})
})
// Add bbox to top level.
Expand Down
Loading