生成图片问题

README.md

@@ -1,72 +1,18 @@
 # TP-GAN
-
-Official TP-GAN Tensorflow implementation for the ICCV17 paper "[Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis](http://openaccess.thecvf.com/content_ICCV_2017/papers/Huang_Beyond_Face_Rotation_ICCV_2017_paper.pdf)" by [Huang, Rui](http://www.andrew.cmu.edu/user/ruih2/) and Zhang, Shu and Li, Tianyu and [He, Ran](http://www.nlpr.ia.ac.cn/english/irds/People/rhe.html).
-
-The goal is to **recover a frontal face image of the same person from a single face image under any poses**.
-
-Here are some examples from the paper.![image](images/ownsynthesis.jpg)
-
-### Testing images
-
-Synthesized  testing images of all poses, corresponding illumination in Setting 2 (and its cropped input) in MultiPIE can be obtained here [Google Drive](https://drive.google.com/file/d/1Kx0sMjFTzLX3-rZ03TAVBAj-gcd9rJrd/view?usp=sharing). 
-
-FAQ: **Synthesized**(not the original) images for other illumination condition and/or training set can be obtained upon request. Unfortunately, I cannot redistribute the original dataset due to copyright. If you would like to access the original MultiPIE dataset, please contact [MultiPIE](http://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html).
-
-### Random examples
-
-Here are random examples of 10 testing image pairs for each degree.
-
-15 and 30 degrees: 
-<p align="center">
-<img src="images/15-rand.png", width="300", border="10">   <img src="images/30-rand.png", width="300", border="10">
-</p> 
-
-45 and 60 degrees:
-<p align="center">
-<img src="images/45-rand.png", width="300", border="10">   <img src="images/60-rand.png", width="304", border="10">
-</p> 
-
-75 and 90 degrees:
-<p align="center">
-<img src="images/75-rand.png", width="300", border="10">   <img src="images/90-rand.png", width="306", border="10">
-</p> 
-
-### Note
-
-It was initially written in Tensorflow 0.12. If you have implemented another version, I'll be happy to reference it here.
-
-This is an initial release of code, which may not be fully tested. Refinement, input data example, pre-trained models, and precomputed testing image features will come later.
-
-The input is cropped with the Matlab script `face_db_align_single_custom.m`, which accepts 5 keypoints and outputs a cropped image and transformed keypoints.
-
-Some example cropping outputs is shown in folder `data-example`.
-
-Our 90-degree model only used 45-90 degree images for training. Other models we trained didn't use 90 degree images. 90 degree images' left and right eye patches coincide.
-
-The 5 keypoints can be extracted from off-the-shelf landmark detectors, e.g. 'Zhang et al. Combining Data-driven and Model-driven Methods for Robust Facial Landmark Detection, 2016'. The synthesis performance is similar to using manually labelled keypoints. See released keypoint files below.
-
-We thank Xiang Wu for providing the [face feature network](https://github.com/AlfredXiangWu/face_verification_experiment). We load it as `DeepFace` in the code, the weights are from a custom Light-CNN cafeemodel file. Our implementation borrowed code from the [dcgan repo](https://github.com/carpedm20/DCGAN-tensorflow).
-
-Update 0.11:
-
-- Releasing *5 keypoint locations* for MultiPIE Session 1-4 dataset. Please download from [here](https://drive.google.com/open?id=1bbMH71SNnX09r7kZj7ExqRLTEUHZVkno). Most of the 60-90 degrees images are labelled manually, others come from MTCNNv2 detector. If you like it, please consider citing our paper.
-- Adding `DeepFace168.pickle` weights file for Light-CNN. Please note this is an improved version than the one originally used in the experiment.
-
-### Citation and Contact
-
-If you like our work or find our code useful, welcome to cite our paper!
-
-Any suggestion and/or comment would be valuable. Please send an email to Rui at [email protected] or other authors.
-
-      @InProceedings{Huang_2017_ICCV,
-      author = {Huang, Rui and Zhang, Shu and Li, Tianyu and He, Ran},
-      title = {Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis},
-      booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
-      month = {Oct},
-      year = {2017}
-      }
-
-### License
-
-This code is freely available for free non-commercial use, and may be redistributed under the conditions set by the license. Please, see the [license](https://github.com/HRLTY/TP-GAN/blob/master/LICENSE) for further details. For commercial queries, please contact [Rui Huang](http://www.andrew.cmu.edu/user/ruih2/) and [Ran He](http://www.nlpr.ia.ac.cn/english/irds/People/rhe.html).
-
+data中5pt文件为人脸关键点部位,用mtcnn模型识别,五行分别是左右眼嘴鼻的x和y坐标,代码能在tensorflow0.12版本下跑,不想修改兼容其他版本了  
+mtcnn的模型不能在tf0.12下跑,用5pt文件做下中转  
+没用到face_db_align_single_custom.m  
+tensorflow0.12要求CUDA版本Cuda 8.0 and CudNN 5.1?  
+# 使用镜像运行
+sudo nvidia-docker pull nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04能跑通  
+拉取镜像后需要:  
+1.apt-get update后apt-get install python3-dev python3-pip vim  
+2.再pip3 install tensorflow-gpu==0.12 scipy pillow  
+3.下载代码,放数据集后修改下路径
+4.修改TP_GAN-Mar6FS_0.12.py里面的参数后运行此文件
+### 测试单张图片  
+1.放好图片,修改mtcnn.py里的文件参数后运行,生成.5pt文件  
+2.修改test_pt.py后运行,输出结果到本文件夹  
+## 数据集相关信息见https://github.com/HRLTY/TP-GAN
+记得修改下identical output的数目  
+uitils.py里的scipy.misc的imread和imsave用imageio的imread和imwrite代替(较新版本的scipy.misc去除了这两个功能)

TP_GAN-Mar6FS.py → TP_GAN-Mar6FS_0.12.py

@@ -21,7 +21,7 @@
 from utils import pp, visualize, to_json
 
 import tensorflow as tf
-
+import logging#pt
 #These parameters should provide a good initialization, but if for specific refinement, you can adjust them during training.
 
 ALPHA_ADVER = 2e1
@@ -58,17 +58,17 @@
 flags.DEFINE_integer("epoch", 250, "Epoch to train [25]")
 flags.DEFINE_float("learning_rate", 1e-4, "Learning rate of for adam [0.0002]")
 flags.DEFINE_float("beta1", 0.9, "Momentum term of adam [0.5]")
-flags.DEFINE_integer("train_size", np.inf, "The size of train images [np.inf]")
-flags.DEFINE_integer("batch_size", 20, "The size of batch images [64]")
+flags.DEFINE_integer("train_size", 1156, "The size of train images [np.inf]")
+flags.DEFINE_integer("batch_size", 4, "The size of batch images [64]")
 flags.DEFINE_integer("image_size", 128, "The size of image to use (will be center cropped) [108]")
 flags.DEFINE_integer("output_size", 128, "The size of the output images to produce [64]")
 flags.DEFINE_integer("c_dim", 3, "Dimension of image color. [3]")
 flags.DEFINE_string("dataset", "MultiPIE", "The name of dataset [celebA, mnist, lsun]")
 flags.DEFINE_string("checkpoint_dir", "checkpoint60", "Directory name to save the checkpoints [checkpoint]")
 flags.DEFINE_string("sample_dir", "samples", "Directory name to save the image samples [samples]")
 flags.DEFINE_boolean("is_train", True, "True for training, False for testing [False]")
-flags.DEFINE_boolean("is_crop", False, "True for training, False for testing [False]")
-flags.DEFINE_boolean("visualize", False, "True for visualizing, False for nothing [False]")
+flags.DEFINE_boolean("is_crop", True, "True for training, False for testing [False]")
+flags.DEFINE_boolean("visualize", True, "True for visualizing, False for nothing [False]")
 FLAGS = flags.FLAGS
 
 class DCGAN(object):
@@ -91,11 +91,11 @@ def __init__(self, sess, image_size=128, is_crop=True,
             c_dim: (optional) Dimension of image color. For grayscale input, set to 1. [3]
         """
         self.test_batch_size = batch_size
-        self.save_interval = 300
+        self.save_interval = 500
         self.sample_interval = 150
         self.sess = sess
         self.is_grayscale = (c_dim == 1)
-        self.batch_size = 10
+        self.batch_size = FLAGS.batch_size
         self.sample_run_num = 15
         self.testing = False
         self.testingphase = 'FS'
@@ -120,11 +120,13 @@ def __init__(self, sess, image_size=128, is_crop=True,
 
         # batch normalization : deals with poor initialization helps gradient flow
         random.seed()
-        self.DeepFacePath = '/home/shu.zhang/ruihuang/data/DeepFace.pickle'
+        self.DeepFacePath = './DeepFace168.pickle'
         self.dataset_name = dataset_name
         self.checkpoint_dir = checkpoint_dir
-        self.loadDeepFace(self.DeepFacePath)
-        self.build_model()
+        self.loadDeepFace()#pt
+        self.build_model()#pt
+
+
 
     def build_model(self):
 
@@ -222,6 +224,7 @@ def build_model(self):
             _,_,_,_, self.G_pool5, self.Gvector = self.FeatureExtractDeepFace(tf.reduce_mean(self.G, axis=3, keep_dims=True))
             _,_,_,_, self.label_pool5, self.labelvector = self.FeatureExtractDeepFace(tf.reduce_mean(self.g_labels, axis=3, keep_dims=True), reuse=True)
             _,_,_,_, _, self.samplevector = self.FeatureExtractDeepFace(tf.reduce_mean(self.sample_images_nocode, axis=3, keep_dims=True), reuse=True)
+
             #self.Dv, self.Dv_logits = self.discriminatorClassify(self.Gvector)
             #self.dv_loss = tf.reduce_sum(tf.nn.sparse_softmax_cross_entropy_with_logits(self.Dv_logits, self.verify_labels))
             self.dv_loss = tf.reduce_mean(tf.abs(self.Gvector-self.labelvector))
@@ -266,7 +269,6 @@ def build_model(self):
         errcheck32 = tf.abs(self.check_sel32 - self.g32_labels) #* mask32
         errcheck64 = tf.abs(self.check_sel64 - self.g64_labels) #* mask64
         errcheck128 = tf.abs(self.check_sel128 - self.g_labels) #* mask128
-
         self.weightedErrL1 = tf.reduce_mean(tf.reduce_sum(tf.reduce_sum(errL1, 1), 1))
         self.symErrL1 = tf.reduce_mean(tf.reduce_sum(tf.reduce_sum(symL1(#self.processor(self.G)
             tf.nn.avg_pool(self.G, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
@@ -275,7 +277,6 @@ def build_model(self):
         self.symErrL2 = tf.reduce_mean(tf.reduce_sum(tf.reduce_sum(symL1(self.processor(self.G2)), 1), 1))
         self.weightedErrL3 = tf.reduce_mean(tf.reduce_sum(tf.reduce_sum(errL1_3, 1), 1))
         self.symErrL3 = tf.reduce_mean(tf.reduce_sum(tf.reduce_sum(symL1(self.processor(self.G3, reuse=True)), 1), 1))
-
         cond_L12 = tf.abs(tf.image.resize_bilinear(self.G, [64,64]) - tf.stop_gradient(self.G2))
         #self.condErrL12 = tf.reduce_mean(tf.reduce_sum(tf.reduce_sum(cond_L12, 1), 1))
         #cond_L23 = tf.abs(tf.image.resize_bilinear(self.G2, [32,32]) - tf.stop_gradient(self.G3))
@@ -293,14 +294,15 @@ def build_model(self):
         self.g_loss = L1_1_W * (self.weightedErrL1 + SYM_W * self.symErrL1) + L1_2_W * (self.weightedErrL2 + SYM_W * self.symErrL2) \
                       + L1_3_W * (self.weightedErrL3 + SYM_W * self.symErrL3)
         self.g_loss += BELTA_FEATURE * self.dv_loss + ALPHA_ADVER * self.g_loss_adver + IDEN_W * self.idenloss + self.tv_loss * TV_WEIGHT
-
         self.rot_loss = PART_W * (self.eyel_loss + self.eyer_loss + self.nose_loss + self.mouth_loss)
         #self.sel_loss = self.weightedErr_check32 + self.weightedErr_check64 + self.weightedErr_check128
         #self.g_loss += self.sel_loss
-
         self.var_file = open('var_log.txt', mode='a')
         t_vars = [var for var in tf.trainable_variables() if 'FeatureExtractDeepFace' not in var.name \
                                                                                     and 'processor' not in var.name]
+        print('done!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!')
+
+
         def isTargetVar(name, tokens):
             for token in tokens:
                 if token in name:
@@ -329,13 +331,14 @@ def isTargetVar(name, tokens):
 
         self.saver = tf.train.Saver(t_vars, max_to_keep=2)
 
+
     def train(self, config):
         """Train DCGAN"""
         #data = glob(os.path.join("./data", config.dataset, "*.jpg"))
         data = MultiPIE(LOAD_60_LABEL=LOAD_60_LABEL, GENERATE_MASK=USE_MASK, RANDOM_VERIFY=RANDOM_VERIFY, MIRROR_TO_ONE_SIDE = True, source = self.testingphase)
         #np.random.shuffle(data)
         config.sample_dir += '{:05d}'.format(random.randint(1,100000))
-
+        #print(self.__dict__,66666666666)
         d_optim = tf.train.AdamOptimizer(config.learning_rate, beta1=config.beta1) \
                           .minimize(self.d_loss, var_list=self.d_vars)
         #clip_D = [p.assign(tf.clip_by_value(p, -CLIP_D, CLIP_D)) for p in self.d_vars]
@@ -361,9 +364,11 @@ def train(self, config):
             print(" [*] Load SUCCESS")
         else:
             print(" [!] Load failed...")
-
+        print(555555555555555555555,self.test_batch_size,self.sample_run_num)
+        #self.test_batch_size * self.sample_run_num分别是10,15
         sample_images, filenames ,sample_eyel, sample_eyer, sample_nose, sample_mouth, \
             sample_labels, sample_leyel, sample_leyer, sample_lnose, sample_lmouth, sample_iden = data.test_batch(self.test_batch_size * self.sample_run_num)
+
         if not self.testing:
             sample_imagesT, filenamesT ,sample_eyelT, sample_eyerT, sample_noseT, sample_mouthT, \
             sample_labelsT, sample_leyelT, sample_leyerT, sample_lnoseT, sample_lmouthT, sample_idenT = data.test_batch(self.test_batch_size * self.sample_run_num * RANK_MUL, Pose=RANGE)
@@ -775,20 +780,27 @@ def evaluate(self,epoch, idx, batch_idxs, start_time, mode,
         self.f.flush()
         print(tobePrint)
     #DEEPFACE MODEL BEGINS---
-    def loadDeepFace(self, DeepFacePath):
+    def loadDeepFace(self, DeepFacePath= None):
         if DeepFacePath is None:
             path = sys.modules[self.__class__.__module__].__file__
             # print path
             path = os.path.abspath(os.path.join(path, os.pardir))
             # print path
-            path = os.path.join(path, "DeepFace.pickle")
+            path = os.path.join(path, "DeepFace168.pickle")
             DeepFacePath = path
             logging.info("Load npy file from '%s'.", DeepFacePath)
         if not os.path.isfile(DeepFacePath):
             logging.error(("File '%s' not found. "), DeepFacePath)
             sys.exit(1)
-        with open(DeepFacePath,'r') as file:
-            self.data_dict = pickle.load(file)
+
+        #with open(DeepFacePath,'r') as file:
+        #    self.data_dict = pickle.load(file)
+        with open(DeepFacePath,'rb') as file: # Binary read
+            u = pickle._Unpickler(file)
+            u.encoding = 'latin1'
+            p = u.load()
+            self.data_dict = p
+
         print("Deep Face pickle data file loaded")
 
     def FeatureExtractDeepFace(self, images, name = "FeatureExtractDeepFace", reuse=False):
@@ -798,7 +810,7 @@ def FeatureExtractDeepFace(self, images, name = "FeatureExtractDeepFace", reuse=
                 scope.reuse_variables()
 
             conv1 = self._conv_layer(images, name='conv1')
-            print(3, type(3))
+            #print(3, type(3))
             slice1_1, slice1_2 = tf.split(3, 2, conv1)
             eltwise1 = tf.maximum(slice1_1, slice1_2)
             pool1 = tf.nn.max_pool(eltwise1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],

data/read_csv.py

@@ -0,0 +1,12 @@
+import csv
+import os
+import numpy as np
+PATH = '/home/ubuntu3000/pt/TP-GAN/data/45_5pt'
+trans_points = np.empty([5,2],dtype=np.int32)
+for filename in os.listdir(PATH):
+    with open(os.path.join(PATH,filename), 'r') as csvfile:
+        reader = csv.reader(csvfile, delimiter=' ')
+        for ind,row in enumerate(reader):
+            trans_points[ind,:] = row 
+        print(trans_points)
+        break

data/write_csv.py

@@ -0,0 +1,19 @@
+import csv
+import os
+
+def create_csv(path_):
+    PATH = '/home/ubuntu3000/pt/TP-GAN/data/45'
+    data=[]
+    data_test=[]
+    path=os.path.join(path_,'train.csv')
+    with open(path, "w") as file:
+        csv_file = csv.writer(file)
+        for filename in os.listdir(PATH):
+            head = [filename]
+            if not 'test' in filename:
+                data.append(head)
+        csv_file.writerows(data)
+
+if __name__== '__main__':
+
+    create_csv('/home/ubuntu3000/pt/TP-GAN/data/')

mtcnn.py

@@ -0,0 +1,88 @@
+import argparse,cv2
+#from mtcnn import MTCNN
+import tensorflow as tf
+import os
+import csv
+class MTCNN:
+
+    def __init__(self, model_path, min_size=40, factor=0.709, thresholds=[0.6, 0.7, 0.7]):
+        self.min_size = min_size
+        self.factor = factor
+        self.thresholds = thresholds
+
+        graph = tf.Graph()
+        with graph.as_default():
+            with open(model_path, 'rb') as f:
+                graph_def = tf.GraphDef.FromString(f.read())
+                tf.import_graph_def(graph_def, name='')
+        self.graph = graph
+        config = tf.ConfigProto(
+            allow_soft_placement=True,
+            intra_op_parallelism_threads=4,
+            inter_op_parallelism_threads=4)
+        config.gpu_options.allow_growth = True
+        self.sess = tf.Session(graph=graph, config=config)
+
+    def detect(self, img):
+        feeds = {
+            self.graph.get_operation_by_name('input').outputs[0]: img,
+            self.graph.get_operation_by_name('min_size').outputs[0]: self.min_size,
+            self.graph.get_operation_by_name('thresholds').outputs[0]: self.thresholds,
+            self.graph.get_operation_by_name('factor').outputs[0]: self.factor
+        }
+        fetches = [self.graph.get_operation_by_name('prob').outputs[0],
+                  self.graph.get_operation_by_name('landmarks').outputs[0],
+                  self.graph.get_operation_by_name('box').outputs[0]]
+        prob, landmarks, box = self.sess.run(fetches, feeds)
+        return box, prob, landmarks
+
+def test_image(PATH):
+    mtcnn = MTCNN('./mtcnn.pb')
+    save_PATH='/home/ubuntu3000/pt/TP-GAN/data/45_5pt'
+
+    for imgpath in os.listdir(PATH):
+        path = os.path.join(save_PATH,imgpath.replace('.png','.5pt'))
+        csvfile = open(path, "w")
+        img = cv2.imread(os.path.join(PATH,imgpath))
+        data=[]
+        bbox, scores, landmarks = mtcnn.detect(img)
+        for box, pts in zip(bbox, landmarks):
+            pts = pts.astype('int32')
+            for i in range(5):
+                row = str(pts[i+5]) + ' '+ str(pts[i])+'\n'
+                csvfile.write(row)
+        csvfile.close()
+
+def create_img(PATH):
+    mtcnn = MTCNN('./mtcnn.pb')
+    csvfile = open('test_tem.5pt', "w")
+    img_rec = cv2.imread(PATH)
+    data=[]
+    bbox, scores, landmarks = mtcnn.detect(img_rec)
+    for box, pts in zip(bbox, landmarks):
+        bbox=bbox.astype('int32')
+        pts = pts.astype('int32')
+        print([int(box[0]),int(box[2]),int(box[1]),int(box[3])])
+        for i in range(5):
+            row = str(pts[i+5]) + ' '+ str(pts[i])+'\n'
+            #img=cv2.circle(img,(pts[i+5],pts[i]),1,(0,255,0),2)
+    img_rec=img_rec[int(box[0])-20:int(box[2])+20,int(box[1])-50:int(box[3])+50]
+    img_rec=cv2.resize(img_rec,(128,128))
+    cv2.imwrite('test_tem.png',img_rec)
+    bbox, scores, landmarks = mtcnn.detect(img_rec)
+    for box, pts in zip(bbox, landmarks):
+        pts = pts.astype('int32')
+        for i in range(5):
+            row = str(pts[i+5]) + ' '+ str(pts[i])+'\n'
+            img_vis=cv2.circle(img_rec,(pts[i+5],pts[i]),1,(0,255,0),2)
+            csvfile.write(row)
+    cv2.imwrite('test_tem_vis.png',img_vis)
+    csvfile.close()
+
+def cut(PATH):
+    img = cv2.imread(PATH)
+    #img=img[]
+    cv2.imwrite('test_tem.png',img)
+
+if __name__ == '__main__':
+    create_img('index.jpeg')