Project 5: Xueyin Wan

README.md

@@ -3,26 +3,126 @@ WebGL Deferred Shading
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Xueyin Wan
+* Platform: *FireFox 49.0.2* on: Windows 10, i7-4870 @ 2.50GHz 16GB, NVIDIA GeForce GT 750M 2GB (Personal Laptop)
+
+###Features I Implemented
+
+* Deferred Blinn-Phong shading
+* Bloom using blur
+* Bloom using two-pass Gaussian blur (extract, blur, blurtwice, combine)
+* Toon Shading (ramp shading, edge detection)
+* Scissor Test Optimization (with debug view)
+* Screen-space Motion Blur 
+* G-Buffer Optimization
 
 ### Live Online
+####Press the .png file below and you may find my demo for this project!! Hope you enjoy :)
+[![](result/intro_to_video.PNG)](https://vimeo.com/190819701)
+Link: https://vimeo.com/190819701
+
+## Showcase My Result
+### Part One
+
+### 1. Deferred Blinn-Phong shading
+|  Depth Map  | Position Map |
+|------|------|
+|![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/deferred-1478436556417.png "Depth Map") | ![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/deferred-1478436560546.png "Position Map") |
+
+| Geometry Normal  | Normal map |
+|------|------|
+|![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/deferred-1478436569856.png "Geometry Normal") | ![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/deferred-1478436581008.png "Normal Map") |
+
+| Surface Normal  | Color map |
+|------|------|
+|![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/deferred-1478436585799.png "Surface Normal") | ![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/deferred-1478436575930.png "Color Map") |
+
+Above pictures show the whole procudure of deferred shading. In our code base, Blinn-Phong shading was the first basic feature. We do not contain any post processing (fancy) effects. Just applied Blinn-Phong lighting in blinnphong-pointlight.frag.glsl and render with this shader.
+
+### 2. Two-pass Gaussian blur Bloom Effect
+|  Without Bloom Effect | With Bloom Effect |
+|------|------|
+|![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/without_bloom.gif "Without Bloom Effect") | ![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/only_bloom.gif " With Bloom Effect") |
+I learned bloom effect on this website: http://learnopengl.com/#!Advanced-Lighting/Bloom
+
+We can see, that basically 4 steps contained in bloom effect. We just follow the steps to achieve this effect.
+
+`First : Extract`
+In this step, we extract all the fragments that exceed a certain brightness.This gives us an image that only shows the bright colored regions as their fragment intensities exceeded a certain threshold.
+
+`Second : Blur Horizontally`
+In this step, we simply took the average of surrounding pixels of an image. We choose Gaussian Operator here. 
 
-[![](img/thumb.png)](http://TODO.github.io/Project5B-WebGL-Deferred-Shading)
+First, we select the horizontal neighbor pixels to start blur.
+We save this first blur in the first framebuffer, since we need to blur it vertically based on this current result.
 
-### Demo Video/GIF
+`Third: Blur Vertically`
+In this step, we toggle to the vertically blur mode.
 
-[![](img/video.png)](TODO)
+With these two steps, we simply finish the Two-pass Gaussian blur and we can save current image to next step.
 
-### (TODO: Your README)
+`Fourth: Combine`
+We combine the result derived after above three steps with our original picture. Then we could get the bloom effect.
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+By the way, I think two pass blur is really clever since we deduplicate a lot of repeated I/Os and calculations.
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+### 3. Toon Shading
+|  Without Toon Shading | With Toon Shading |
+|------|------|
+|![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/without-toon-shading.gif "Without Toon Shading") | ![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/toon-shading.gif " With Toon Shading") |
 
+Toon shading contains two steps: ramp shading + edge detection.
+In ramp shading procedure, what we need to think about is that: the diffuse and specular values in the blinn-phong shading should be calculated based on step functions, rather than continous functions. So we use step function to classify the fragments with different lambert and specular terms.
+
+After we finish ramp shading, we come to the edge detection stage. Luckily I've using opencv during the year 2014 and 2015, so I have some basic image processing concepts about this. Basically here, we select our contour based on depth information of the fragment itself and surrounding pixels.
+
+### 4. Motion Blur
+| Without Motion Blur | With Motion Blur |
+|------|------|
+|![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/without_motion_blur.gif "Without Motion Blur") | ![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/motion_blur_correct2.gif "With Motion Blur") |
+
+Here, we need to take advantage of camera's projection matrix since in our scene, in fact here it is the camera always moving, not the geometry in the scene. We then need the previous frame's camera projection matrix(we could save it as a global variable). Then we come into next frame, we first pass this to camera's prev_matrix property, then using current frame's camera projection matrix to update this global previous projection matrix. This method works fine!
+
+How do I think out this idea? Thanks to this very very very useful tutorial: http://http.developer.nvidia.com/GPUGems3/gpugems3_ch27.html
+
+In this tutorial, we could see that we need the previous camera projection matrix information. Then I just follow the formula in the code provided in this tutorial to get the correct result.
+
+We could get the blur vector between two frames, and simply divide by 2 to get the velocity.(blur along velocity direction)
+
+### Part Two: Optimization
+### 1. Scissor Test
+| Debug View Of Scissor Test | Debug View Of Scissor Test |
+|------|------|
+|![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/debug_scissor.gif "Debug Scissor Test") | ![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/debug_scissor2.gif "Debug Scissor Test") |
+With scissor test, we do not need to render the full image. We only need to render a rectangle area around the light. This will definitely shorten the time of rendering.
+Let's see a chart to judge the performance of Scissor test.
+![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/Scissor%20charts.PNG "Scissor Charts")
+We could clearly see that, with scissor test, the render procedure each frame reaches up to 3 times when scissor test off.
+
+### 2. Decrease the size of the G-Buffer
+When read through the code, I found that
+```javascript
+    vec3 pos = gb0.xyz;     // World-space position
+    vec3 geomnor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
+    vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
+    vec3 normap = gb3.xyz;  // The raw normal map (normals relative to the surface they're on)
+    vec3 nor = applyNormalMap (geomnor, normap); // Surface normal   
+```
+We can see in the code above, we have gb1 and gb3 contain two vectors related to normal : geometry normal and raw normal map. Since at last we need to apply Normal map to Geometry normal and finally get a surface normal, we could save the spaces and do the computation in between, and then directly pass Surface normal to gbuffer. It definitely won't affect the correctness of the result but SAVE SPACE and IMPROVE TIME EFFICIENCY.
+
+Now the code chaged to 
+```javascript
+    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
+    vec4 gb2 = texture2D(u_gbufs[2], v_uv);
+    float depth = texture2D(u_depth, v_uv).x;
+    vec3 pos = gb0.xyz;     // World-space position
+    vec3 nor = normalize(gb1.xyz); // Combine
+    vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
+```
+And the optimization results are in the chart below.
+![alt text](https://github.com/xueyinw/Project5-WebGL-Deferred-Shading-with-glTF/blob/master/result/GBuffer%20charts.PNG "G-Buffer Optimization")
+We can see, as we increased light numbers, when we still has gb0 - gb3, the time to render one frame is 35ms, 50ms, 66ms. If we reduced the number of GBuffer and has only gb0 - gb2, the time to render one frame is 27ms, 38ms, 52ms. With such results we can say that the time efficiency has been improved about 1.3 times.
 
 ### Credits
 
@@ -31,3 +131,8 @@ to implementation work. Complete the implementation early to leave time!
 * [webgl-debug](https://github.com/KhronosGroup/WebGLDeveloperTools) by Khronos Group Inc.
 * [glMatrix](https://github.com/toji/gl-matrix) by [@toji](https://github.com/toji) and contributors
 * [minimal-gltf-loader](https://github.com/shrekshao/minimal-gltf-loader) by [@shrekshao](https://github.com/shrekshao)
+
+### Extra Credits
+* [Toon Shading](https://en.wikibooks.org/wiki/GLSL_Programming/Unity/Toon_Shading)
+* [Bloom Effect](http://learnopengl.com/#!Advanced-Lighting/Bloom)
+* [Motion Blur](http://http.developer.nvidia.com/GPUGems3/gpugems3_ch27.html)

glsl/clear.frag.glsl

@@ -3,7 +3,8 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+//#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
 
 void main() {
     for (int i = 0; i < NUM_GBUFFERS; i++) {

glsl/copy.frag.glsl

@@ -10,11 +10,31 @@ varying vec3 v_position;
 varying vec3 v_normal;
 varying vec2 v_uv;
 
+/*
+vec3 pos = gb0.xyz;     // World-space position
+vec3 geomnor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
+vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
+vec3 normap = gb3.xyz;  // The raw normal map (normals relative to the surface they're on)
+vec3 nor = applyNormalMap (geomnor, normap);     // The true normals as we want to light them - with the normal map applied to the geometry normals (applyNormalMap above)
+*/
+//reduce a normal!
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
 void main() {
     // TODO: copy values into gl_FragData[0], [1], etc.
     // You can use the GLSL texture2D function to access the textures using
     // the UV in v_uv.
 
     // this gives you the idea
-    // gl_FragData[0] = vec4( v_position, 1.0 );
+    gl_FragData[0] = vec4( v_position, 1.0 );
+  //  gl_FragData[1] = vec4( v_normal, 1.0 );
+    gl_FragData[1] = vec4(applyNormalMap(v_normal, texture2D(u_normap, v_uv).rgb), 1.0);  //conbine normal together
+    gl_FragData[2] = texture2D(u_colmap, v_uv);
+  //  gl_FragData[3] = texture2D(u_normap, v_uv);
 }

glsl/deferred/ambient.frag.glsl

@@ -3,25 +3,28 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
-
+//#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
 varying vec2 v_uv;
 
 void main() {
-    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
-    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
+//    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+//    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+//    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
 
+    vec3 colmap = gb2.xyz;
+
     if (depth == 1.0) {
         gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
         return;
     }
-
-    gl_FragColor = vec4(0.1, 0.1, 0.1, 1);  // TODO: replace this
+    gl_FragColor = vec4(0.2 * colmap,1.0);
+// Ambient color!
+//    gl_FragColor = vec4(0.1, 0.1, 0.1, 1);  // TODO: replace this
 }

glsl/deferred/blinnphong-pointlight.frag.glsl

@@ -2,17 +2,18 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
-
+//#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
 uniform vec3 u_lightCol;
 uniform vec3 u_lightPos;
 uniform float u_lightRad;
+uniform vec3 u_camPos; // add camera position
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
-
+uniform float u_toon;
 varying vec2 v_uv;
 
-vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) { //normal map -> geometry normal, each geometry get its normal through this way
     normap = normap * 2.0 - 1.0;
     vec3 up = normalize(vec3(0.001, 1, 0.001));
     vec3 surftan = normalize(cross(geomnor, up));
@@ -24,9 +25,22 @@ void main() {
     vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+//    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
+    // local? extract, alright
+
+     vec3 pos = gb0.xyz;
+  //   vec3 geomnor = gb1.xyz;
+     vec3 color = gb2.rgb;
+//     vec3 normap = gb3.xyz;
+     vec3 nor = normalize(gb1.xyz);
+
+     vec3 lightDir = normalize(u_lightPos - pos);
+     float distToLight = distance(u_lightPos, pos);
+     vec3 reflDir = reflect(-lightDir, nor);
+     vec3 viewDir = normalize(u_camPos - pos);
+     vec3 halfDir = normalize(lightDir + viewDir);
 
     // If nothing was rendered to this pixel, set alpha to 0 so that the
     // postprocessing step can render the sky color.
@@ -35,5 +49,31 @@ void main() {
         return;
     }
 
-    gl_FragColor = vec4(0, 0, 1, 1);  // TODO: perform lighting calculations
+    float attenuation = max(0.0, u_lightRad - length(pos - u_lightPos));
+//    float attenuation = 1.0 / (1.0 + u_lightRad * distToLight * distToLight);
+    float lambert = max(dot(lightDir, nor),0.0);
+  	float specular = 0.0;
+  	if(lambert > 0.0)	{
+  		float ndotH = max(dot(halfDir,nor),0.0);
+  		specular = pow(ndotH, 12.0);
+  	}
+    vec3 finalcolor = (lambert * color + specular) * u_lightCol * attenuation;
+    float ndotH = max(dot(halfDir,nor),0.0);
+
+    if (u_toon > 0.6) { //let's make a threshold
+       if (lambert < 0.6) {
+          lambert = 0.2;
+       } else {
+          lambert = 1.0;
+       }
+       if (ndotH > 0.75) {
+          specular = 1.0; //pow(ndotH, 12.0);
+       } else {
+          specular = 0.0;
+       }
+    finalcolor = (lambert * color + specular) * u_lightCol * attenuation;
+    }
+
+    gl_FragColor = vec4(finalcolor, 1.0);
+//    gl_FragColor = vec4(0, 0, 1, 1);  // TODO: perform lighting calculations
 }

glsl/deferred/debug.frag.glsl

@@ -2,12 +2,18 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
-
+//#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 3
 uniform int u_debug;
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
+uniform mat4 u_prevProj;
+uniform vec3 u_camPos;
+
+//http://http.developer.nvidia.com/GPUGems3/gpugems3_ch27.html
+//http://john-chapman-graphics.blogspot.com/2013/01/what-is-motion-blur-motion-pictures-are.html
+//Help! Motion Blur.....
 varying vec2 v_uv;
 
 const vec4 SKY_COLOR = vec4(0.66, 0.73, 1.0, 1.0);
@@ -24,29 +30,36 @@ void main() {
     vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+  //  vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
     // These definitions are suggested for starting out, but you will probably want to change them.
     vec3 pos = gb0.xyz;     // World-space position
-    vec3 geomnor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
+  //  vec3 geomnor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
     vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
-    vec3 normap = gb3.xyz;  // The raw normal map (normals relative to the surface they're on)
-    vec3 nor = applyNormalMap (geomnor, normap);     // The true normals as we want to light them - with the normal map applied to the geometry normals (applyNormalMap above)
+//    vec3 normap = gb3.xyz;  // The raw normal map (normals relative to the surface they're on)
+    vec3 nor = normalize(gb1.xyz);//applyNormalMap (geomnor, normap);     // The true normals as we want to light them - with the normal map applied to the geometry normals (applyNormalMap above)
+
+// H is the viewport position at this pixel in the range -1 to 1.
+    vec4 cur_pos = vec4(v_uv.x * 2.0 - 1.0, (1.0 - v_uv.y) * 2.0 - 1.0, depth, 1.0);
+    vec4 prev_pos =  u_prevProj * vec4(pos / gb0.w, 1.0);
+    prev_pos /= prev_pos.w;
+    vec2 velocity = (cur_pos.xy - prev_pos.xy) / 2.0;
 
     // TODO: uncomment
     if (u_debug == 0) {
         gl_FragColor = vec4(vec3(depth), 1.0);
     } else if (u_debug == 1) {
-        // gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
+        gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
     } else if (u_debug == 2) {
-        // gl_FragColor = vec4(abs(geomnor), 1.0);
+        gl_FragColor = vec4(abs(nor), 1.0);//vec4(abs(geomnor), 1.0);
     } else if (u_debug == 3) {
-        // gl_FragColor = vec4(colmap, 1.0);
-    } else if (u_debug == 4) {
-        // gl_FragColor = vec4(normap, 1.0);
-    } else if (u_debug == 5) {
-        // gl_FragColor = vec4(abs(nor), 1.0);
+        gl_FragColor = vec4(colmap, 1.0);
+    // } else if (u_debug == 4) {
+    //     gl_FragColor = vec4(normap, 1.0);
+    // } else if (u_debug == 5) {
+    //    gl_FragColor = vec4(abs(nor), 1.0);
+    // }
     } else {
         gl_FragColor = vec4(1, 0, 1, 1);
     }

glsl/deferred/scissor.frag.glsl

@@ -0,0 +1,7 @@
+#version 100
+precision highp float;
+precision highp int;
+
+void main() {
+    gl_FragColor = vec4(0.5, 0.0, 0.6, 0.1);
+}