[Feature Request][Spark] Auto Compaction shouldn't be trigged if compaction hasn't been run yet

[mwc360]

Feature request

The OSS implementation will run compaction when auto-compaction is enabled if compaction hasn't been run yet. I.e. running a CTAS w/ the table property enabled will perform compaction after the write even if the small file count doesn't meet minNumFiles.

delta/spark/src/main/scala/org/apache/spark/sql/delta/stats/AutoCompactPartitionStats.scala

Lines 71 to 78 in a920885

	/**
	* Determine whether this partition can be autocompacted based on the number of small files or
	* if this [[AutoCompactPartitionStats]] instance has not auto compacted it yet.
	* @param minNumFiles The minimum number of files this table-partition should have to trigger
	* Auto Compaction in case it has already been compacted once.
	*/
	def hasSufficientSmallFilesOrHasNotBeenCompacted(minNumFiles: Long): Boolean =
	!wasAutoCompacted || hasSufficientFiles(minNumFiles)

Auto compaction in Databricks does not perform this unnecessary initial compaction operation. It should only be evaluated based on the presence of small files which meet or exceed the minNumFiles.

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Overview

Motivation

Improve performance of tables that get created with auto compaction enabled.

Further details

Willingness to contribute

• Yes. I can contribute this feature independently.
• Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
• No. I cannot contribute this feature at this time.

[mwc360]

@nicklan - what was the intent behind always triggering auto compact when the feature is enabled? Auto compact in DBX doesn't work this same way.

i.e. running a CTAS w/ the autoCompact property runs optimize right after the CTAS completes, even when the count of small files is well below minNumFiles.