Use Amazon S3 Inventory to list objects

[maouw]

Currently, s3bids relies on the file listing capabilities of s3fs, which incurs additional costs.

Amazon S3 Inventory can help reduce these costs in addition to making it easier to quickly match regular expressions to paths/keys:

Amazon S3 Inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or objects with a shared prefix (that is, objects that have names that begin with a common string). If you set up a weekly inventory, a report is generated every Sunday (UTC time zone) after the initial report. For information about Amazon S3 Inventory pricing, see Amazon S3 pricing.

Although there are some limitations -- the minimum interval for reports is 1 day -- this still could be very helpful, especially if it is possible to list only changes since the time of the last report.