HELP – train SAEHD or AMP error

[giu3232Participant]

Hi guys,

I’m very new to deepfake. I’m using DeepFaceLab_NVIDIA_RTX3000_series_build_11_20_2021. I managed to gather around 4200 pictures of the guy I d like to deepfake, I have all the heads aligned properly, guides were checked one by one and are ok, and I trained the masks using Xseg ( created around 200 manually).

When I use Quick96 it works, but when I try with SAEH or AMP I get the same error.

Here is when the console tells me:

Running trainer.

[new] No saved models found. Enter a name of a new model : WAYNE_KNIGHT_SAEHD
WAYNE_KNIGHT_SAEHD

Model first run.

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
[0] : NVIDIA GeForce RTX 4090

[0] Which GPU indexes to choose? : 0
0

[16] Autobackup every N hour ( 0..24 ?:help ) : 1
1
[y] Write preview history ( y/n ?:help ) :
y
[n] Choose image for the preview history ( y/n ) :
n
[1000000] Target iteration : 0
0
[n] Flip SRC faces randomly ( y/n ?:help ) :
n
[n] Flip DST faces randomly ( y/n ?:help ) :
n
[8] Batch_size ( ?:help ) : 12
12
[512] Resolution ( 64-640 ?:help ) : 2048
2048
[head] Face type ( h/mf/f/wf/head ?:help ) :
head
[liae-ud] AE architecture ( ?:help ) : df-ud
df-ud
[512] AutoEncoder dimensions ( 32-1024 ?:help ) : 2048
2048
[64] Encoder dimensions ( 16-256 ?:help ) : 256
256
[64] Decoder dimensions ( 16-256 ?:help ) : 256
256
[22] Decoder mask dimensions ( 16-256 ?:help ) : 256
256
[n] Masked training ( y/n ?:help ) : y
[y] Eyes and mouth priority ( y/n ?:help ) :
y
[y] Uniform yaw distribution of samples ( y/n ?:help ) : n
[y] Blur out mask ( y/n ?:help ) :
y
[y] Place models and optimizer on GPU ( y/n ?:help ) :
y
[y] Use AdaBelief optimizer? ( y/n ?:help ) :
y
[y] Use learning rate dropout ( n/y/cpu ?:help ) :
y
[y] Enable random warp of samples ( y/n ?:help ) :
y
[0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) : 0.05
0.05
[0.0] GAN power ( 0.0 .. 5.0 ?:help ) : 0.01
0.01
[64] GAN patch size ( 3-640 ?:help ) : 512
512
[16] GAN dimensions ( 4-512 ?:help ) :
16
[0.0] ‘True face’ power. ( 0.0000 .. 1.0 ?:help ) : 0.01
0.01
[0.0] Face style power ( 0.0..100.0 ?:help ) : 0
0.0
[0.0] Background style power ( 0.0..100.0 ?:help ) : 0
0.0
[none] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) : rct
rct
[n] Enable gradient clipping ( y/n ?:help ) : y
[n] Enable pretraining mode ( y/n ?:help ) :
n
Error: Dimension 0 in both shapes must be equal, but are 5 and 6. Shapes are [5,5] and [6,6]. for ‘{{node concat_4}} = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32](LeakyRelu_81, LeakyRelu_84, concat_4/axis)’ with input shapes: [?,512,5,5], [?,512,6,6], [] and with computed input tensors: input[2] = <1>.
Traceback (most recent call last):
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py”, line 1880, in _create_c_op
c_op = pywrap_tf_session.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 0 in both shapes must be equal, but are 5 and 6. Shapes are [5,5] and [6,6]. for ‘{{node concat_4}} = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32](LeakyRelu_81, LeakyRelu_84, concat_4/axis)’ with input shapes: [?,512,5,5], [?,512,6,6], [] and with computed input tensors: input[2] = <1>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\mainscripts\Trainer.py”, line 58, in trainerThread
debug=debug)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\models\ModelBase.py”, line 193, in __init__
self.on_initialize()
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\models\Model_SAEHD\Model.py”, line 518, in on_initialize
gpu_pred_src_src_d2 = self.D_src(gpu_pred_src_src_masked_opt)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\leras\models\ModelBase.py”, line 117, in __call__
return self.forward(*args, **kwargs)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\DeepFaceLab\core\leras\models\PatchDiscriminator.py”, line 184, in forward
x = tf.concat( [enc, x], axis=nn.conv2d_ch_axis)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py”, line 206, in wrapper
return target(*args, **kwargs)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\array_ops.py”, line 1769, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_array_ops.py”, line 1227, in concat_v2
“ConcatV2”, values=values, axis=axis, name=name)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py”, line 750, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py”, line 3569, in _create_op_internal
op_def=op_def)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py”, line 2042, in __init__
control_input_ops, op_def)
File “T:\DEEPFAKE\DeepFaceLab_NVIDIA_RTX3000_series\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py”, line 1883, in _create_c_op
raise ValueError(str(e))
ValueError: Dimension 0 in both shapes must be equal, but are 5 and 6. Shapes are [5,5] and [6,6]. for ‘{{node concat_4}} = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32](LeakyRelu_81, LeakyRelu_84, concat_4/axis)’ with input shapes: [?,512,5,5], [?,512,6,6], [] and with computed input tensors: input[2] = <1>.

Any chance someone here knows where are the wrong parameters? I tried to let all the default value too and still get the same error. I tried to change tons of parameters, going down on resolution, going from df to liae … I tried the rtx3000 serie and dx12 version of the software, nothing works.

export AMP and SAEHD as dfm both return an error about a 2gig limit too.

Thanks a lot for your help.

[Author]

Posts