Reply To: SAED does not work

Home Forums DeepFaceLab Training SAED does not work Reply To: SAED does not work

#3563
aron
Participant

    Thanks for your answer.
    I implemented the first two suggestions, but still got an error message.
    What do you mean by the last point: “Might also try increasing your page file size.”
    Can you explain that in a bit more detail?
    Best regards
    aron

    Running trainer.

    [new] No saved models found. Enter a name of a new model :
    new

    Model first run.

    Choose one or several GPU idxs (separated by comma).

    [CPU] : CPU
    [0] : NVIDIA GeForce GTX 950M

    [0] Which GPU indexes to choose? : 0
    0

    [0] Autobackup every N hour ( 0..24 ?:help ) :
    0
    [n] Write preview history ( y/n ?:help ) :
    n
    [0] Target iteration : 10
    10
    [n] Flip SRC faces randomly ( y/n ?:help ) :
    n
    [y] Flip DST faces randomly ( y/n ?:help ) :
    y
    [4] Batch_size ( ?:help ) : 2
    2
    [128] Resolution ( 64-640 ?:help ) : 64
    64
    [f] Face type ( h/mf/f/wf/head ?:help ) :
    f
    [liae-ud] AE architecture ( ?:help ) :
    liae-ud
    [256] AutoEncoder dimensions ( 32-1024 ?:help ) :
    256
    [64] Encoder dimensions ( 16-256 ?:help ) :
    64
    [64] Decoder dimensions ( 16-256 ?:help ) :
    64
    [22] Decoder mask dimensions ( 16-256 ?:help ) :
    22
    [n] Eyes and mouth priority ( y/n ?:help ) :
    n
    [n] Uniform yaw distribution of samples ( y/n ?:help ) :
    n
    [n] Blur out mask ( y/n ?:help ) :
    n
    [y] Place models and optimizer on GPU ( y/n ?:help ) : n
    [y] Use AdaBelief optimizer? ( y/n ?:help ) : n
    [n] Use learning rate dropout ( n/y/cpu ?:help ) :
    n
    [y] Enable random warp of samples ( y/n ?:help ) :
    y
    [0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) :
    0.0
    [0.0] GAN power ( 0.0 .. 5.0 ?:help ) :
    0.0
    [0.0] Face style power ( 0.0..100.0 ?:help ) :
    0.0
    [0.0] Background style power ( 0.0..100.0 ?:help ) :
    0.0
    [none] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) :
    none
    [n] Enable gradient clipping ( y/n ?:help ) :
    n
    [n] Enable pretraining mode ( y/n ?:help ) :
    n
    Initializing models: 100%|###############################################################| 5/5 [00:03<00:00, 1.64it/s]
    Loading samples: 100%|###############################################################| 461/461 [00:38<00:00, 11.91it/s]
    Loading samples: 100%|###############################################################| 399/399 [00:35<00:00, 11.09it/s]
    Process Process-13:
    Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py”, line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 145, in process
    img = get_eyes_mouth_mask()*mask
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 78, in get_eyes_mouth_mask
    mouth_mask = LandmarksProcessor.get_image_mouth_mask (sample_bgr.shape, sample_landmarks)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\facelib\LandmarksProcessor.py”, line 447, in get_image_mouth_mask
    hull_mask = cv2.dilate(hull_mask, cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(dilate,dilate)), iterations = 1 )
    cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\alloc.cpp:55: error: (-4:Insufficient memory) Failed to allocate 16777216 bytes in function ‘cv::OutOfMemoryError’

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “multiprocessing\process.py”, line 258, in _bootstrap
    File “multiprocessing\process.py”, line 93, in run
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py”, line 54, in process_func
    gen_data = next (self.generator_func)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py”, line 136, in batch_func
    raise Exception (“Exception occured in sample %s. Error: %s” % (sample.filename, traceback.format_exc() ) )
    Exception: Exception occured in sample C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\workspace\data_dst\aligned\00229_0.jpg. Error: Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py”, line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 145, in process
    img = get_eyes_mouth_mask()*mask
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 78, in get_eyes_mouth_mask
    mouth_mask = LandmarksProcessor.get_image_mouth_mask (sample_bgr.shape, sample_landmarks)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\facelib\LandmarksProcessor.py”, line 447, in get_image_mouth_mask
    hull_mask = cv2.dilate(hull_mask, cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(dilate,dilate)), iterations = 1 )
    cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\alloc.cpp:55: error: (-4:Insufficient memory) Failed to allocate 16777216 bytes in function ‘cv::OutOfMemoryError’

    ================== Model Summary ===================
    == ==
    == Model name: new_SAEHD ==
    == ==
    == Current iteration: 0 ==
    == ==
    ==—————- Model Options —————–==
    == ==
    == resolution: 64 ==
    == face_type: f ==
    == models_opt_on_gpu: False ==
    == archi: liae-ud ==
    == ae_dims: 256 ==
    == e_dims: 64 ==
    == d_dims: 64 ==
    == d_mask_dims: 22 ==
    == masked_training: True ==
    == eyes_mouth_prio: False ==
    == uniform_yaw: False ==
    == blur_out_mask: False ==
    == adabelief: False ==
    == lr_dropout: n ==
    == random_warp: True ==
    == random_hsv_power: 0.0 ==
    == true_face_power: 0.0 ==
    == face_style_power: 0.0 ==
    == bg_style_power: 0.0 ==
    == ct_mode: none ==
    == clipgrad: False ==
    == pretrain: False ==
    == autobackup_hour: 0 ==
    == write_preview_history: False ==
    == target_iter: 10 ==
    == random_src_flip: False ==
    == random_dst_flip: True ==
    == batch_size: 2 ==
    == gan_power: 0.0 ==
    == gan_patch_size: 8 ==
    == gan_dims: 16 ==
    == ==
    ==—————— Running On ——————==
    == ==
    == Device index: 0 ==
    == Name: NVIDIA GeForce GTX 950M ==
    == VRAM: 1.35GB ==
    == ==
    ====================================================
    Starting. Target iteration: 10. Press “Enter” to stop training and save model.

    Trying to do the first iteration. If an error occurs, reduce the model parameters.

    !!!
    Windows 10 users IMPORTANT notice. You should set this setting in order to work correctly.

    View post on imgur.com


    !!!
    You are training the model from scratch. It is strongly recommended to use a pretrained model to speed up the training and improve the quality.

    Error: Could not allocate ndarray
    Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1334, in _do_call
    return fn(*args)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1407, in _call_tf_sessionrun
    run_metadata)
    tensorflow.python.framework.errors_impl.InternalError: Could not allocate ndarray

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\mainscripts\Trainer.py”, line 159, in trainerThread
    model_save()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\mainscripts\Trainer.py”, line 68, in model_save
    model.save()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\models\ModelBase.py”, line 393, in save
    self.onSave()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\models\Model_SAEHD\Model.py”, line 759, in onSave
    model.save_weights ( self.get_strpath_storage_for_file(filename) )
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\leras\layers\Saveable.py”, line 51, in save_weights
    w_val = nn.tf_sess.run (w).copy()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 929, in run
    run_metadata_ptr)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1152, in _run
    feed_dict_tensor, options, run_metadata)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1328, in _do_run
    run_metadata)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1348, in _do_call
    raise type(e)(node_def, op, message)
    tensorflow.python.framework.errors_impl.InternalError: Could not allocate ndarray

    Running trainer.

    [new] No saved models found. Enter a name of a new model :
    new

    Model first run.

    Choose one or several GPU idxs (separated by comma).

    [CPU] : CPU
    [0] : NVIDIA GeForce GTX 950M

    [0] Which GPU indexes to choose? : 0
    0

    [0] Autobackup every N hour ( 0..24 ?:help ) :
    0
    [n] Write preview history ( y/n ?:help ) :
    n
    [0] Target iteration : 10
    10
    [n] Flip SRC faces randomly ( y/n ?:help ) :
    n
    [y] Flip DST faces randomly ( y/n ?:help ) :
    y
    [4] Batch_size ( ?:help ) : 2
    2
    [128] Resolution ( 64-640 ?:help ) : 64
    64
    [f] Face type ( h/mf/f/wf/head ?:help ) :
    f
    [liae-ud] AE architecture ( ?:help ) :
    liae-ud
    [256] AutoEncoder dimensions ( 32-1024 ?:help ) :
    256
    [64] Encoder dimensions ( 16-256 ?:help ) :
    64
    [64] Decoder dimensions ( 16-256 ?:help ) :
    64
    [22] Decoder mask dimensions ( 16-256 ?:help ) :
    22
    [n] Eyes and mouth priority ( y/n ?:help ) :
    n
    [n] Uniform yaw distribution of samples ( y/n ?:help ) :
    n
    [n] Blur out mask ( y/n ?:help ) :
    n
    [y] Place models and optimizer on GPU ( y/n ?:help ) : n
    [y] Use AdaBelief optimizer? ( y/n ?:help ) : n
    [n] Use learning rate dropout ( n/y/cpu ?:help ) :
    n
    [y] Enable random warp of samples ( y/n ?:help ) :
    y
    [0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) :
    0.0
    [0.0] GAN power ( 0.0 .. 5.0 ?:help ) :
    0.0
    [0.0] Face style power ( 0.0..100.0 ?:help ) :
    0.0
    [0.0] Background style power ( 0.0..100.0 ?:help ) :
    0.0
    [none] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) :
    none
    [n] Enable gradient clipping ( y/n ?:help ) :
    n
    [n] Enable pretraining mode ( y/n ?:help ) :
    n
    Initializing models: 100%|###############################################################| 5/5 [00:03<00:00, 1.64it/s]
    Loading samples: 100%|###############################################################| 461/461 [00:38<00:00, 11.91it/s]
    Loading samples: 100%|###############################################################| 399/399 [00:35<00:00, 11.09it/s]
    Process Process-13:
    Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py”, line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 145, in process
    img = get_eyes_mouth_mask()*mask
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 78, in get_eyes_mouth_mask
    mouth_mask = LandmarksProcessor.get_image_mouth_mask (sample_bgr.shape, sample_landmarks)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\facelib\LandmarksProcessor.py”, line 447, in get_image_mouth_mask
    hull_mask = cv2.dilate(hull_mask, cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(dilate,dilate)), iterations = 1 )
    cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\alloc.cpp:55: error: (-4:Insufficient memory) Failed to allocate 16777216 bytes in function ‘cv::OutOfMemoryError’

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “multiprocessing\process.py”, line 258, in _bootstrap
    File “multiprocessing\process.py”, line 93, in run
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py”, line 54, in process_func
    gen_data = next (self.generator_func)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py”, line 136, in batch_func
    raise Exception (“Exception occured in sample %s. Error: %s” % (sample.filename, traceback.format_exc() ) )
    Exception: Exception occured in sample C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\workspace\data_dst\aligned\00229_0.jpg. Error: Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py”, line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 145, in process
    img = get_eyes_mouth_mask()*mask
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 78, in get_eyes_mouth_mask
    mouth_mask = LandmarksProcessor.get_image_mouth_mask (sample_bgr.shape, sample_landmarks)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\facelib\LandmarksProcessor.py”, line 447, in get_image_mouth_mask
    hull_mask = cv2.dilate(hull_mask, cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(dilate,dilate)), iterations = 1 )
    cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\alloc.cpp:55: error: (-4:Insufficient memory) Failed to allocate 16777216 bytes in function ‘cv::OutOfMemoryError’

    ================== Model Summary ===================
    == ==
    == Model name: new_SAEHD ==
    == ==
    == Current iteration: 0 ==
    == ==
    ==—————- Model Options —————–==
    == ==
    == resolution: 64 ==
    == face_type: f ==
    == models_opt_on_gpu: False ==
    == archi: liae-ud ==
    == ae_dims: 256 ==
    == e_dims: 64 ==
    == d_dims: 64 ==
    == d_mask_dims: 22 ==
    == masked_training: True ==
    == eyes_mouth_prio: False ==
    == uniform_yaw: False ==
    == blur_out_mask: False ==
    == adabelief: False ==
    == lr_dropout: n ==
    == random_warp: True ==
    == random_hsv_power: 0.0 ==
    == true_face_power: 0.0 ==
    == face_style_power: 0.0 ==
    == bg_style_power: 0.0 ==
    == ct_mode: none ==
    == clipgrad: False ==
    == pretrain: False ==
    == autobackup_hour: 0 ==
    == write_preview_history: False ==
    == target_iter: 10 ==
    == random_src_flip: False ==
    == random_dst_flip: True ==
    == batch_size: 2 ==
    == gan_power: 0.0 ==
    == gan_patch_size: 8 ==
    == gan_dims: 16 ==
    == ==
    ==—————— Running On ——————==
    == ==
    == Device index: 0 ==
    == Name: NVIDIA GeForce GTX 950M ==
    == VRAM: 1.35GB ==
    == ==
    ====================================================
    Starting. Target iteration: 10. Press “Enter” to stop training and save model.

    Trying to do the first iteration. If an error occurs, reduce the model parameters.

    !!!
    Windows 10 users IMPORTANT notice. You should set this setting in order to work correctly.

    View post on imgur.com


    !!!
    You are training the model from scratch. It is strongly recommended to use a pretrained model to speed up the training and improve the quality.

    Error: Could not allocate ndarray
    Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1334, in _do_call
    return fn(*args)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1407, in _call_tf_sessionrun
    run_metadata)
    tensorflow.python.framework.errors_impl.InternalError: Could not allocate ndarray

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\mainscripts\Trainer.py”, line 159, in trainerThread
    model_save()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\mainscripts\Trainer.py”, line 68, in model_save
    model.save()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\models\ModelBase.py”, line 393, in save
    self.onSave()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\models\Model_SAEHD\Model.py”, line 759, in onSave
    model.save_weights ( self.get_strpath_storage_for_file(filename) )
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\leras\layers\Saveable.py”, line 51, in save_weights
    w_val = nn.tf_sess.run (w).copy()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 929, in run
    run_metadata_ptr)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1152, in _run
    feed_dict_tensor, options, run_metadata)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1328, in _do_run
    run_metadata)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1348, in _do_call
    raise type(e)(node_def, op, message)
    tensorflow.python.framework.errors_impl.InternalError: Could not allocate ndarray

    Running trainer.

    [new] No saved models found. Enter a name of a new model :
    new

    Model first run.

    Choose one or several GPU idxs (separated by comma).

    [CPU] : CPU
    [0] : NVIDIA GeForce GTX 950M

    [0] Which GPU indexes to choose? : 0
    0

    [0] Autobackup every N hour ( 0..24 ?:help ) :
    0
    [n] Write preview history ( y/n ?:help ) :
    n
    [0] Target iteration : 10
    10
    [n] Flip SRC faces randomly ( y/n ?:help ) :
    n
    [y] Flip DST faces randomly ( y/n ?:help ) :
    y
    [4] Batch_size ( ?:help ) : 2
    2
    [128] Resolution ( 64-640 ?:help ) : 64
    64
    [f] Face type ( h/mf/f/wf/head ?:help ) :
    f
    [liae-ud] AE architecture ( ?:help ) :
    liae-ud
    [256] AutoEncoder dimensions ( 32-1024 ?:help ) :
    256
    [64] Encoder dimensions ( 16-256 ?:help ) :
    64
    [64] Decoder dimensions ( 16-256 ?:help ) :
    64
    [22] Decoder mask dimensions ( 16-256 ?:help ) :
    22
    [n] Eyes and mouth priority ( y/n ?:help ) :
    n
    [n] Uniform yaw distribution of samples ( y/n ?:help ) :
    n
    [n] Blur out mask ( y/n ?:help ) :
    n
    [y] Place models and optimizer on GPU ( y/n ?:help ) : n
    [y] Use AdaBelief optimizer? ( y/n ?:help ) : n
    [n] Use learning rate dropout ( n/y/cpu ?:help ) :
    n
    [y] Enable random warp of samples ( y/n ?:help ) :
    y
    [0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) :
    0.0
    [0.0] GAN power ( 0.0 .. 5.0 ?:help ) :
    0.0
    [0.0] Face style power ( 0.0..100.0 ?:help ) :
    0.0
    [0.0] Background style power ( 0.0..100.0 ?:help ) :
    0.0
    [none] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) :
    none
    [n] Enable gradient clipping ( y/n ?:help ) :
    n
    [n] Enable pretraining mode ( y/n ?:help ) :
    n
    Initializing models: 100%|###############################################################| 5/5 [00:03<00:00, 1.64it/s]
    Loading samples: 100%|###############################################################| 461/461 [00:38<00:00, 11.91it/s]
    Loading samples: 100%|###############################################################| 399/399 [00:35<00:00, 11.09it/s]
    Process Process-13:
    Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py”, line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 145, in process
    img = get_eyes_mouth_mask()*mask
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 78, in get_eyes_mouth_mask
    mouth_mask = LandmarksProcessor.get_image_mouth_mask (sample_bgr.shape, sample_landmarks)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\facelib\LandmarksProcessor.py”, line 447, in get_image_mouth_mask
    hull_mask = cv2.dilate(hull_mask, cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(dilate,dilate)), iterations = 1 )
    cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\alloc.cpp:55: error: (-4:Insufficient memory) Failed to allocate 16777216 bytes in function ‘cv::OutOfMemoryError’

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “multiprocessing\process.py”, line 258, in _bootstrap
    File “multiprocessing\process.py”, line 93, in run
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py”, line 54, in process_func
    gen_data = next (self.generator_func)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py”, line 136, in batch_func
    raise Exception (“Exception occured in sample %s. Error: %s” % (sample.filename, traceback.format_exc() ) )
    Exception: Exception occured in sample C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\workspace\data_dst\aligned\00229_0.jpg. Error: Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py”, line 134, in batch_func
    x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 145, in process
    img = get_eyes_mouth_mask()*mask
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\samplelib\SampleProcessor.py”, line 78, in get_eyes_mouth_mask
    mouth_mask = LandmarksProcessor.get_image_mouth_mask (sample_bgr.shape, sample_landmarks)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\facelib\LandmarksProcessor.py”, line 447, in get_image_mouth_mask
    hull_mask = cv2.dilate(hull_mask, cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(dilate,dilate)), iterations = 1 )
    cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\alloc.cpp:55: error: (-4:Insufficient memory) Failed to allocate 16777216 bytes in function ‘cv::OutOfMemoryError’

    ================== Model Summary ===================
    == ==
    == Model name: new_SAEHD ==
    == ==
    == Current iteration: 0 ==
    == ==
    ==—————- Model Options —————–==
    == ==
    == resolution: 64 ==
    == face_type: f ==
    == models_opt_on_gpu: False ==
    == archi: liae-ud ==
    == ae_dims: 256 ==
    == e_dims: 64 ==
    == d_dims: 64 ==
    == d_mask_dims: 22 ==
    == masked_training: True ==
    == eyes_mouth_prio: False ==
    == uniform_yaw: False ==
    == blur_out_mask: False ==
    == adabelief: False ==
    == lr_dropout: n ==
    == random_warp: True ==
    == random_hsv_power: 0.0 ==
    == true_face_power: 0.0 ==
    == face_style_power: 0.0 ==
    == bg_style_power: 0.0 ==
    == ct_mode: none ==
    == clipgrad: False ==
    == pretrain: False ==
    == autobackup_hour: 0 ==
    == write_preview_history: False ==
    == target_iter: 10 ==
    == random_src_flip: False ==
    == random_dst_flip: True ==
    == batch_size: 2 ==
    == gan_power: 0.0 ==
    == gan_patch_size: 8 ==
    == gan_dims: 16 ==
    == ==
    ==—————— Running On ——————==
    == ==
    == Device index: 0 ==
    == Name: NVIDIA GeForce GTX 950M ==
    == VRAM: 1.35GB ==
    == ==
    ====================================================
    Starting. Target iteration: 10. Press “Enter” to stop training and save model.

    Trying to do the first iteration. If an error occurs, reduce the model parameters.

    !!!
    Windows 10 users IMPORTANT notice. You should set this setting in order to work correctly.

    View post on imgur.com


    !!!
    You are training the model from scratch. It is strongly recommended to use a pretrained model to speed up the training and improve the quality.

    Error: Could not allocate ndarray
    Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1334, in _do_call
    return fn(*args)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1407, in _call_tf_sessionrun
    run_metadata)
    tensorflow.python.framework.errors_impl.InternalError: Could not allocate ndarray

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\mainscripts\Trainer.py”, line 159, in trainerThread
    model_save()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\mainscripts\Trainer.py”, line 68, in model_save
    model.save()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\models\ModelBase.py”, line 393, in save
    self.onSave()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\models\Model_SAEHD\Model.py”, line 759, in onSave
    model.save_weights ( self.get_strpath_storage_for_file(filename) )
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\DeepFaceLab\core\leras\layers\Saveable.py”, line 51, in save_weights
    w_val = nn.tf_sess.run (w).copy()
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 929, in run
    run_metadata_ptr)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1152, in _run
    feed_dict_tensor, options, run_metadata)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1328, in _do_run
    run_metadata)
    File “C:\Users\aronk\Desktop\11.20. DeepFaceLab_NVIDIA_up_to_RTX2080Ti\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py”, line 1348, in _do_call
    raise type(e)(node_def, op, message)
    tensorflow.python.framework.errors_impl.InternalError: Could not allocate ndarray