This repository contains code that aims to detect the lane a car is driving in. It uses OpenCV 3 (term cv2 to refer to it here) and Python 3.
The following Jupyter Notebook needs a python 3 kernel and it runs in a conda environment described in file environment.yml. The Jupyter Notebook was tested in a Mac OS El Capitan.
To create the environment, use your terminal to navigate to the folder that contains the environment.yml file and type the following command:
$conda env create --file environment.yml
The whole process of detecting the lanes is based on computer vision algorithms. The pipeline of such process is as follows:
It reads an image using the module matplotlib.image and if you want to read a video it uses the module VideoFileClip from moviepy.editor.
It changes the picture's current color space to HLS. This is done to help the color segmentation to reduce the sensitivity of color change due to light conditions.
Since lanes are white and yellow, the segmentation needs to consider both colors. One mask for each color in HLS space is created and then both mask are bitwise_OR to result in one mask that filters both colors. Such mask is applied to the original image using a bitwise_AND to have a filtered image.
NOTE: The mask can be applied to the original image as the pixels' location in HLS and RGB/BGR never changed location only values.
Here the color-segmented image is coverted to grayscale for further process
The image still has very sharp edges that can be considered noisy, so smoothing them is the right thing to do before trying to detect edges.
Apply the Canny edge detection algorithm to detect the all edges in the image.
After detecting edges there are still some that the lane detection does not need. Thus, a region of interest (ROI) is used to crete a window around the area of the current lane of the car.
Once the ROI has filtered undesired edges, the Hough Transform is applied to find potential points of lines that constitute the lane lines. You can read more about Hough Transform implementation in OpenCV3 here.
Then, these lines can be mapped to the original picture just for visual confirmation as below. In the pipeline one takes only the points that make such lines.
The set of points returned from the Hough algorithm are used to group the points by slope's sign. The right lane's line will have points with negative slope (cyan) and the left lane's line with positive slope (yellow). It was key to leave out points with slope values between [-0.2, 0.2], as they can be edges that couldn't be filtered and are not part of the lane's line.
The goal is to obtain the line equation for each lane's lines. Since we have two sets of points grouped by slope and two different lines, it can be seen as two datasets for two different equations. Said that it is possible to use Machine learning simple linear regression algorithm to estimate the two line's equations. An important key point in this step is that the images have their origin (0,0) on the top left. This makes it harder for us to use x as the features since the car is not always in the same x while it moves. However, the perspective of the car to see the horizon still the same and that is y. Thus, using y as the features to predict x as car moves gives a better result. In here, the line calculation uses Machine Learning Linear Regression with y as the features and x as the labels. The image below shows that each line was calculated from a different group of points.
Finally, once the line equations are estimated, the equations can be given the maximum value of y in the image to predict the corresponding x and find the points for the lanes on the bottom of the image. Then, the intersection-point of both lines can be calculated and used to determine the y coordinate to use to predict x and find the end point of the lines to draw. These, same point can be used to draw a polygon that fills the are of the lane with a color.
or if you prefer you can only fill in the lane with a color
All images are overlapped using a linear blending.
The code has still a lot of space for improvement and make it more robust. Some improvements that can be made are:
- Implement a robust solution in the current pipeline for night time scenarios
- Handle the cases when there are no points returned from Hough transform
- Implement a distortion step to handle curves better
- Create an algorithm to dynamically reduce the ROI for lanes detection, but keep detecting cars infront in the same lane. This is to avoid miscalculation of lanes
- The current ML linear regression is basically overfitting as it is calculated for each image. However, a smarter way could be to collect all data and train a lane detector algorithm that can be generalized to as many as possible lanes in different locations.
The shortcomings are quite related to improvements in this case, and some of these are:
- Pipeline very dependent in the Hough transform and detection of points in every image
- It does not handle situations when white or yellow cars change lanes infront of the car and contribute to data in the ROI that causes nosie to calculate the lanes.
- It has never been tested at night time
- It considers the camera is calibrated and fixed to one position in the car
- The car is only aware of its lane due to the ROI used
With all of these improvements and shortcomings mentioned, hope one of you get interested in contributing to this repo and create together a more robust and generalized pipeline for lane detection.