Update logging to use Rust tracing_subscriber and

avoid Python logger for Rust components.
Add DEVELOPMENT.md instructions on enabling logging.

CHANGELOG.md

@@ -1,3 +1,12 @@
+## Unreleased
+
+### New features
+
+### Breaking changes
+* Separate completely Rust logs and Python logs. Rust logs are configured through RUST_LOG,
+  S3_CONNECTOR_ENABLE_CRT_LOGS and S3_CONNECTOR_LOGS_DIR_PATH environment variables.
+
+
 ## v1.2.0 (March 13, 2024)
 
 ### New features
@@ -11,6 +20,7 @@
 ### Breaking changes
 * No breaking changes.
 
+
 ## v1.1.4 (February 26, 2024)
 
 ### New features
@@ -37,7 +47,6 @@
 * No breaking changes.
 
 
-
 ## v1.1.2 (January 19, 2024)
 
 ### New features
@@ -48,7 +57,6 @@
 * No breaking changes.
 
 
-
 ## v1.1.1 (December 11, 2023)
 
 ### New features
@@ -69,13 +77,12 @@
 * No breaking changes.
 
 
-
 ## v1.0.0 (November 22, 2023)
 * The Amazon S3 Connector for PyTorch delivers high throughput for PyTorch training jobs that access and store data in Amazon S3.
 
 ### New features
-* S3IterableDataset and S3MapDataset, which allow building either an iterable-style or map-style dataset, using your S3 
-stored data, by specifying an S3 URI (a bucket and optional prefix) and the region the bucket is in.
+* S3IterableDataset and S3MapDataset, which allow building either an iterable-style or map-style dataset, using your S3
+  stored data, by specifying an S3 URI (a bucket and optional prefix) and the region the bucket is in.
 * Support for multiprocess data loading for the above datasets.
 * S3Checkpoint, an interface for saving and loading model checkpoints directly to and from an S3 bucket.
 

doc/DEVELOPMENT.md

@@ -92,3 +92,34 @@ Fill in the path of the Python executable in your virtual environment (`venv/bin
 as the program argument.
 Then put a breakpoint in the Rust/C code and try running it.
 
+#### Enabling Logging
+The Python logger handles the logging messages from the Python implementation. 
+The logs of our Rust components are handled through a 
+[tracing_subscriber](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/). 
+
+You have the following configuration options to filter Rust log messages:
+- Default - If you do not have `RUST_LOG` environment variable set up at all, Rust logging will cover `ERROR` level 
+  of Mountpoint S3 Client Rust component.
+
+- Mountpoint S3 Client logs - Configure [RUST_LOG](https://docs.rs/env_logger/latest/env_logger/#enabling-logging) variable. For example, setting
+  `RUST_LOG=debug` will enable logging of DEBUG messages from Mountpoint S3 Client.
+
+- CRT logs - Configure `S3_CONNECTOR_ENABLE_CRT_LOGS` variable, similarly to 
+  [RUST_LOG](https://docs.rs/env_logger/latest/env_logger/#enabling-logging). This will enable the logs from CRT 
+  component and override the `RUST_LOG` setup. 
+
+  **Please note that these logs are very noisy, do NOT enable them unless otherwise instructed.**
+
+Additionally, you can set up `S3_CONNECTOR_LOGS_DIR_PATH` with the path to a local folder where you have WRITE 
+permissions. When this is set up, the Rust logs will be written at this location, 
+with `s3torchconnectorclient.log` prefix, rolling on an hourly basis. 
+
+Example:
+- Enable Mountpoint S3 Client logs with DEBUG level to be written at `/tmp/s3torchconnector-logs`
+```sh
+  export RUST_LOG=debug
+  export S3_CONNECTOR_LOGS_DIR_PATH="/tmp/s3torchconnector-logs"
+  python ./my_test.py
+```
+After running this, you will find the logs under `/tmp/s3torchconnector-logs`. The logging messages will be appended 
+to the most recent file of format `s3torchconnectorclient.log.YYYY-MM-DD-HH`.

s3torchconnectorclient/Cargo.toml

@@ -19,8 +19,8 @@ built = "0.7"
 pyo3 = { version = "0.19.2" }
 pyo3-log = "0.8.3"
 futures = "0.3.28"
-mountpoint-s3-client = { version = "0.7.0", features = ["mock"] }
-mountpoint-s3-crt = "0.6.1"
+mountpoint-s3-client = { version = "0.8.0", features = ["mock"] }
+mountpoint-s3-crt = "0.6.2"
 log = "0.4.20"
 tracing = { version = "0.1.40", default-features = false, features = ["std", "log"] }
 tracing-subscriber = { version = "0.3.18", features = ["fmt", "env-filter"]}