Semantic Segmentation Models for Foggy Weather (CMU 16-824 project).

This is a Pytorch implementation for training and testing different semantic segmentation models on the cityscapes dataset. We also use foggy weather data generated from the cityscpes dataset, to improve performance of baseline segmentation models in adverse weather.

Models Currently Supported

DeepLab V3

⚙️ Setup

Create a fresh eviornment using Anaconda distribution. You can then install the dependencies with:

conda install pytorch torchvision torchaudio -c pytorch
conda install opencv=4.1.2
conda install matplotlib

Recommended python version: 3.7

💾 Cityscapes Training Data

The dataset required for training and evalution can be downloaded using

sh utils/download_dataset.sh

YOUR_USER_NAME and YOUR_PASSWORD in the script should be replaced with the users credentials.

The data should then be placed in 'data/' directory.

Pre-process the dataset to convert label images to ids, using

python utils/preprocess_data.py PATH_TO_CITYSCAPES_DATASET

Foggy weather data can be downloaded using

gdown https://drive.google.com/uc?id=1jIwFebKGmYuvpiN0EQ5KSYWmXjqStQac
# unzip -q -o leftImg8bit_trainvaltest_foggy.zip
# mv leftImg8bit_foggy cityscapes/

It should be extracted and placed in the same directory as unperturbed data.

🖼️ Prediction on Images

You can predict the sementation labels on unperturbed, and foggy data at all scales using the demo script demo.py. Use the following commands,

mkdir demo
python demo.py

The predictions of class labels will be color coded and saved as images to the demo folder.

⏳ Training

You can train on cityscapes good weather or foggy weather dataset using,

mkdir saved_models
python train.py

The train command will also plot training and validation loss values at every epoch, and save the updated model at the end of each epoch to the saved_models directory.

📊 Evaluation

Computing the class wise confusion matrix gives a good insight into the performance of a classification model. You can compute the confusion matrix for the model you trained using the command,

python metrics/compute_confusion_mat.py --model_path 'path to your model' --fog_scale 'can be 0, 0.005, 0.01, 0.02' --dataset_path 'path to your dataset' '--compute_unperturbed'

compute_unperturbed flag should be used if you wish to compute the confusion matrix for the model trained on good weather data. Two models have been provided in the pretrained models folder, one trained on good weather cityscapes data for 150 epochs, and the other one trained using this pretrained model on foggy data for 20 epochs.

The average precision values for each class can then be computed using,

python metrics/compute_ap.py --conf_mat_path "path to your confusion matrix" --fog_intensity "the fog intensity, can be 0, 0.005, 0.1, 0.2"

References

1. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
   Liang-Chieh Chen+, George Papandreou+, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.
   [link]. TPAMI 2017.

2. Rethinking Atrous Convolution for Semantic Image Segmentation
   Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam.
   [link]. arXiv: 1706.05587, 2017.

3. CityScapes
   Rich metadata: preceding and trailing video frames · stereo · GPS · vehicle odometry.
   [link]

4. DeepLab V3 Pytorch Implementation
   Parts of code have been adopted from this repository.
   [link]