#StackBounty: #python #tensorflow #keras #batch-normalization Keras Batchnormalization and sample weights

Bounty: 50

I am trying the the training and evaluation example on the tensorflow website.
Specifically, this part:

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

x_train = x_train.reshape(60000, 784).astype('float32') / 255
x_test = x_test.reshape(10000, 784).astype('float32') / 255

y_train = y_train.astype('float32')
y_test = y_test.astype('float32')

def get_uncompiled_model():
  inputs = keras.Input(shape=(784,), name='digits')
  x = layers.Dense(64, activation='relu', name='dense_1')(inputs)
  x = layers.BatchNormalization()(x)
  x = layers.Dense(64, activation='relu', name='dense_2')(x)
  outputs = layers.Dense(10, activation='softmax', name='predictions')(x)
  model = keras.Model(inputs=inputs, outputs=outputs)
  return model

def get_compiled_model():
  model = get_uncompiled_model()
  model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
              loss='sparse_categorical_crossentropy',
              metrics=['sparse_categorical_accuracy'])
  return model

sample_weight = np.ones(shape=(len(y_train),))
sample_weight[y_train == 5] = 2.

# Create a Dataset that includes sample weights
# (3rd element in the return tuple).
train_dataset = tf.data.Dataset.from_tensor_slices(
    (x_train, y_train, sample_weight))

# Shuffle and slice the dataset.
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)

model = get_compiled_model()
model.fit(train_dataset, epochs=3)

It appears that if I add the batch normalization layer (this line: x = layers.BatchNormalization()(x)) I get the following error:

InvalidArgumentError: The second input must be a scalar, but it has shape [64]
[[{{node batch_normalization_2/cond/ReadVariableOp/Switch}}]]

Any ideas?


Get this bounty!!!

#StackBounty: #python #numpy #tensorflow #machine-learning #keras Keras predict() returns a better accuracy than evaluate()

Bounty: 50

I set up a model with Keras, then I trained it on a dataset of 3 records and finally I tested the resulting model with evaluate() and predict(), using the same test set for both functions (the test set has 100 records and it doesn’t have any record of the training set, as much as it can be relevant, given the size of the two datasets).
The dataset is composed by 5 files, where 4 files represent each one a different temperature sensor, that each minute collects 60 measurements (each row contains 60 measurements), while the last file contains the class labels that I want to predict (in particular, 3 classes: 3, 20 or 100).

This is the model I’m using:

n_sensors, t_periods = 4, 60

model = Sequential()

model.add(Conv1D(100, 6, activation='relu', input_shape=(t_periods, n_sensors)))

model.add(Conv1D(100, 6, activation='relu'))

model.add(MaxPooling1D(3))

model.add(Conv1D(160, 6, activation='relu'))

model.add(Conv1D(160, 6, activation='relu'))

model.add(GlobalAveragePooling1D())

model.add(Dropout(0.5))

model.add(Dense(3, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

That I train:
self.model.fit(X_train, y_train, batch_size=3, epochs=5, verbose=1)

Then I use evaluate:
self.model.evaluate(x_test, y_test, verbose=1)

And predict:

predictions = self.model.predict(data)
result = np.where(predictions[0] == np.amax(predictions[0]))
if result[0][0] == 0:
    return '3'
elif result[0][0] == 1:
    return '20'
else:
    return '100'

For each class predicted, I confront it with the actual label, and then I calculate correct guesses / total examples, that should be equivalent to accuracy from the evaluate() function. Here’s the code:

correct = 0
for profile in self.profile_file: #profile_file is an opened file
    ts1 = self.ts1_file.readline()
    ts2 = self.ts2_file.readline()
    ts3 = self.ts3_file.readline()
    ts4 = self.ts4_file.readline()
    data = ts1, ts2, ts3, ts4
    test_data = self.dl.transform(data) # see the last block of code I posted
    prediction = self.model.predict(test_data)
    if prediction == label:
       correct += 1
acc = correct / 100 # 100 is the number of total examples

Data feeded to evaluate() is taken from this function:

label = pd.read_csv(os.path.join(self.testDir, 'profile.txt'), sep='t', header=None)
label = np_utils.to_categorical(label[0].factorize()[0])
data = [os.path.join(self.testDir,'TS2.txt'),os.path.join(self.testDir, 'TS1.txt'),os.path.join(self.testDir,'TS3.txt'),os.path.join(self.testDir, 'TS4.txt')]
df = pd.DataFrame()
for txt in data:
    read_df = pd.read_csv(txt, sep='t', header=None)
    df = df.append(read_df)
df = df.apply(self.__predict_scale)
df = df.sort_index().values.reshape(-1,4,60).transpose(0,2,1)
return df, label

While data feeded to predict() is taken from this other one:

df = pd.DataFrame()
for txt in data: # data 
    read_df = pd.read_csv(StringIO(txt), sep='t', header=None)
    df = df.append(read_df)
df = df.apply(self.__predict_scale)
df = df.sort_index().values.reshape(-1,4,60).transpose(0,2,1)
return df

Accuracies yielded by evaluate() and predict() are always different: in particular, the maximum difference I noted was when evaluate() resulted in a 78% accuracy while predict() in a 95% accuracy. The only difference between the two functions is that I make predict() work on an example at a time, while evaluate() takes the entire dataset all at once, but it should result in no difference. How can it be?

UPDATE 1: It seems that the problem is in how I prepare my data. In the case of predict(), I transform only one line at a time from each file using the last block of code I posted, while in feeding evaluate(), I transform the entire files using the other function reported. Why should it be different? It seems to me that I’m applying the exact same transformation, the only difference is in the number of rows transformed.


Get this bounty!!!

#StackBounty: #tensorflow How to add after each iteration in tensorflow

Bounty: 50

I am trying to achieve the following:

compute the losses in the previous 25 predictions and sum them before
computing the gradient. I have tried this:

loss_summation=tf.Variable(0,dtype=tf.dtypes.float32,name="loss")
xentropy=tf.nn.sparse_softmax_cross_entropy_with_logits(labels=next_element[1],logits=logits2,name="xentropy")
loss=tf.math.reduce_sum(tf.reduce_mean(xentropy,name="loss"))
loss_summation=tf.assign(loss_summation,loss_summation+loss)
optimizer = tf.train.AdamOptimizer(learning_rate=self.learning_rate)
 gvs = optimizer.compute_gradients(loss_summation,[vars])
with tf.Session() as sess():
      for i in range(25):
                     b=sess.run([loss_summation])

However optimizer.compute_gradients() complains that
None values not supported. How can go around this ?


Get this bounty!!!

#StackBounty: #neural-network #keras #tensorflow #training #ensemble-modeling Training an ensemble of small neural networks efficiently…

Bounty: 50

I have a bunch of small neural networks (say, 5 to 50 feed-forward neural networks with only two hidden layers with 10-100 neurons each), which differ only in the weight initialization. I want to train them all on the same, smallish dataset (say, 10K rows), with a batch size of 1. The aim of this is to combine them into an ensemble by averaging the results.

Now, of course I can build the whole ensemble as one neural network in TensorFlow/Keras, like this:

def bagging_ensemble(inputs: int, width: int, weak_learners: int):
    r'''Return a generic dense network model

    inputs: number of columns (features) in the input data set
    width: number of neurons in the hidden layer of each weak learner
    weak_learners: number of weak learners in the ensemble
    '''
    assert width >= 1, 'width is required to be at least 1'
    assert weak_learners >= 1, 'weak_learners is required to be at least 1'

    activation = tf.keras.activations.tanh
    kernel_initializer = tf.initializers.GlorotUniform()

    input_layer = tf.keras.Input(shape=(inputs,))
    layers = input_layer
    hidden = tf.keras.layers.Dense(units=width, activation=activation, kernel_initializer=kernel_initializer)
                (input_layer)
    hidden = []
    # add hidden layer as a list of weak learners
    for i in range(weak_learners):
        weak_learner = tf.keras.layers.Dense(units=width, activation=activation, kernel_initializer=kernel_initializer)
                        (input_layer)
        weak_learner = tf.keras.layers.Dense(units=1, activation=tf.keras.activations.sigmoid)(weak_learner)
        hidden.append(weak_learner)

    output_layer = tf.keras.layers.Average()(hidden)  # add an averaging layer at the end

    return tf.keras.Model(input_layer, output_layer)      

example_model = bagging_ensemble(inputs=30, width=10, weak_learners=5)
tf.keras.utils.plot_model(example_model)

The resulting model’s plot looks like this:
Plot of Keras model

However, training the model is slow, and because of the batch size of 1, a GPU doesn’t really speed up the process. How can I make better use of the GPU when training such a model in TensorFlow 2, without using a larger batch size?

[The motivation for using this kind of ensemble is the following: Especially with small datasets, each of the small networks will yield different results because of different random initializations. By bagging as demonstrated here, the quality of the resulting model is greatly enhanced. If you’re interested in the thorough neural network modelling approach this technique comes from, look for the works of H. G. Zimmermann.]


Get this bounty!!!

#StackBounty: #tensorflow How to use a loss functions closed-form gradient even when it does not have a closed-form itself

Bounty: 50

I have a loss function without a closed analytic form that I want to implement in tensorflow that has a nice clean gradient (i.e. i have some nice gradient function $f(x)$ that i want to use for backprop but $int_x f(x) dx$ is with no closed form)

Ofcourse i wouldnt be able to evaluate the loss, but i want to use $f(x)$ in the optimization process. How could this be accomplished?


Get this bounty!!!

#StackBounty: #python #keras #tensorflow tflite_convert a Keras h5 model which has a custom loss function results in a ValueError, even…

Bounty: 100

I have written a SRGAN implementation. In the entry point class of the Python program, I declare a function which returns a mean square using the VGG19 model:

# <!--- COST FUNCTION --->
def build_vgg19_loss_network(ground_truth_image, predicted_image):
    loss_model = Vgg19Loss.define_loss_model(high_resolution_shape)
    return mean(square(loss_model(ground_truth_image) - loss_model(predicted_image)))

import keras.losses
keras.losses.build_vgg19_loss_network = build_vgg19_loss_network
# <!--- /COST FUNCTION --->

(Vgg19Loss class shown further below)

As you can see, I have added this custom loss function in the import keras.losses. Why? Because I thought it could solve the following problem…: When I execute the command tflite_convert --output_file=srgan.tflite --keras_model_file=srgan.h5, the Python interpreter raises this error:

raise ValueError(‘Unknown ‘ + printable_module_name + ‘:’ + object_name)
ValueError: Unknown loss function:build_vgg19_loss_network

However, it didn’t solve the problem. Any other solution which could work?

Here is the Vgg19Loss class:

from keras import Model
from keras.applications import VGG19


class Vgg19Loss:
    def __init__(self):
        pass

    @staticmethod
    def define_loss_model(high_resolution_shape):
        model_vgg19 = VGG19(False, 'imagenet', input_shape=high_resolution_shape)
        model_vgg19.trainable = False
        for l in model_vgg19.layers:
            l.trainable = False
        loss_model = Model(model_vgg19.input, model_vgg19.get_layer('block5_conv4').output)
        loss_model.trainable = False
        return loss_model


Get this bounty!!!

#StackBounty: #python #tensorflow #keras #loss-function Joint training of multi-output Keras model

Bounty: 50

I’m writing two joint decoders in Keras, with one common input, two separate outputs, and a loss function that takes both outputs into account.

Here is the minimal Keras code that you can regenerate the error:

import tensorflow as tf
from scat import *

from keras.layers import Input, Reshape, Permute, Lambda, Flatten
from keras.layers.core import Dense
from keras.layers.advanced_activations import LeakyReLU
from keras.models import Model
from keras import backend as K

def identity(x):
    return K.identity(x)

# custom loss function
def custom_loss():
    def my_loss(y_dummy, pred):
        fcn_loss_1 = tf.nn.softmax_cross_entropy_with_logits(labels=y_dummy[0], logits=pred[0])
        fcn_loss_2 = tf.nn.softmax_cross_entropy_with_logits(labels=y_dummy[1], logits=pred[1])
        fcn_loss_2 = tf.matrix_band_part(fcn_loss_2, 0, -1) - tf.matrix_band_part(fcn_loss_2, 0, 0)

        fcn_loss = tf.reduce_mean(fcn_loss_1) + 2 * tf.reduce_mean(fcn_loss_2)

        return fcn_loss
    return my_loss

def keras_version():
    input = Input(shape=(135,), name='feature_input')
    out1 = Dense(128, kernel_initializer='glorot_normal', activation='linear')(input)
    out1 = LeakyReLU(alpha=.2)(out1)
    out1 = Dense(256, kernel_initializer='glorot_normal', activation='linear')(out1)
    out1 = LeakyReLU(alpha=.2)(out1)
    out1 = Dense(512, kernel_initializer='glorot_normal', activation='linear')(out1)
    out1 = LeakyReLU(alpha=.2)(out1)
    out1 = Dense(45, kernel_initializer='glorot_normal', activation='linear')(out1)
    out1 = LeakyReLU(alpha=.2)(out1)
    out1 = Reshape((9, 5))(out1)

    out2 = Dense(128, kernel_initializer='glorot_normal', activation='linear')(input)
    out2 = LeakyReLU(alpha=.2)(out2)
    out2 = Dense(256, kernel_initializer='glorot_normal', activation='linear')(out2)
    out2 = LeakyReLU(alpha=.2)(out2)
    out2 = Dense(512, kernel_initializer='glorot_normal', activation='linear')(out2)
    out2 = LeakyReLU(alpha=.2)(out2)
    out2 = Dense(540, kernel_initializer='glorot_normal', activation='linear')(out2)
    out2 = LeakyReLU(alpha=.2)(out2)
    out2 = Reshape((9, 4, 15))(out2)
    out2 = Lambda(lambda x: K.dot(K.permute_dimensions(x, (0, 2, 1, 3)),
                                  K.permute_dimensions(x, (0, 2, 3, 1))), output_shape=(4,9,9))(out2)
    out2 = Flatten()(out2)
    out2 = Dense(324, kernel_initializer='glorot_normal', activation='linear')(out2)
    out2 = LeakyReLU(alpha=.2)(out2)
    out2 = Reshape((4, 9, 9))(out2)
    out2 = Lambda(lambda x: K.permute_dimensions(x, (0, 2, 3, 1)))(out2)

    out1 = Lambda(identity, name='output_1')(out1)
    out2 = Lambda(identity, name='output_2')(out2)

    return Model(input, [out1, out2])

model = keras_version()
model.compile(loss=custom_loss(), optimizer='adam')

model.summary()

feature_final = np.random.normal(0,1,[5000, 9, 15])
train_features_array = np.random.normal(0,1,[5000, 9, 5])
train_adj_array = np.random.normal(0,1,[5000, 9, 9, 4])

feature_final = feature_final.reshape(-1, 135)
model.fit(feature_final, [train_features_array, train_adj_array],
                batch_size=50,
                epochs=10
                )

The error I get is:

File "...", line 135, in <module>
    epochs=10
File ".../keras/engine/training.py", line 1039, in fit
    validation_steps=validation_steps)
File ".../keras/backend/tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
File ".../tensorflow/python/client/session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: input must be at least 2-dim, received shape: [9]
     [[{{node loss/output_1_loss/MatrixBandPart_1}}]]

On a second attempt, I tried writing two loss functions and using loss weights to combine them.

# custom loss function
def custom_loss_1():
    def my_loss_1(y_dummy, pred):
        fcn_loss_1 = tf.nn.softmax_cross_entropy_with_logits(labels=y_dummy[0], logits=pred[0])

        return tf.reduce_mean(fcn_loss_1)
    return my_loss_1

def custom_loss_2():
    def my_loss_2(y_dummy, pred):
        fcn_loss_2 = tf.nn.softmax_cross_entropy_with_logits(labels=y_dummy[1], logits=pred[1])
        fcn_loss_2 = tf.matrix_band_part(fcn_loss_2, 0, -1) - tf.matrix_band_part(fcn_loss_2, 0, 0)

        return tf.reduce_mean(fcn_loss_2)
    return my_loss_2

model.compile(loss={'output_1':custom_loss_1(), 'output_2':custom_loss_2()},
              loss_weights={'output_1':1.0, 'output_2':2.0}, optimizer='adam')

but I received

tensorflow.python.framework.errors_impl.InvalidArgumentError: Matrix size-incompatible: In[0]: [20,25920], In[1]: [324,324]
     [[{{node dense_9/BiasAdd}}]]

In that case, it might actually be from the model itself. Here is the model.summary:

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
feature_input (InputLayer)      (None, 135)          0                                            
__________________________________________________________________________________________________
dense_5 (Dense)                 (None, 128)          17408       feature_input[0][0]              
__________________________________________________________________________________________________
leaky_re_lu_5 (LeakyReLU)       (None, 128)          0           dense_5[0][0]                    
__________________________________________________________________________________________________
dense_6 (Dense)                 (None, 256)          33024       leaky_re_lu_5[0][0]              
__________________________________________________________________________________________________
leaky_re_lu_6 (LeakyReLU)       (None, 256)          0           dense_6[0][0]                    
__________________________________________________________________________________________________
dense_7 (Dense)                 (None, 512)          131584      leaky_re_lu_6[0][0]              
__________________________________________________________________________________________________
leaky_re_lu_7 (LeakyReLU)       (None, 512)          0           dense_7[0][0]                    
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 128)          17408       feature_input[0][0]              
__________________________________________________________________________________________________
dense_8 (Dense)                 (None, 540)          277020      leaky_re_lu_7[0][0]              
__________________________________________________________________________________________________
leaky_re_lu_1 (LeakyReLU)       (None, 128)          0           dense_1[0][0]                    
__________________________________________________________________________________________________
leaky_re_lu_8 (LeakyReLU)       (None, 540)          0           dense_8[0][0]                    
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 256)          33024       leaky_re_lu_1[0][0]              
__________________________________________________________________________________________________
reshape_2 (Reshape)             (None, 9, 4, 15)     0           leaky_re_lu_8[0][0]              
__________________________________________________________________________________________________
leaky_re_lu_2 (LeakyReLU)       (None, 256)          0           dense_2[0][0]                    
__________________________________________________________________________________________________
lambda_1 (Lambda)               (None, 4, 9, 9)      0           reshape_2[0][0]                  
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 512)          131584      leaky_re_lu_2[0][0]              
__________________________________________________________________________________________________
flatten_1 (Flatten)             (None, 324)          0           lambda_1[0][0]                   
__________________________________________________________________________________________________
leaky_re_lu_3 (LeakyReLU)       (None, 512)          0           dense_3[0][0]                    
__________________________________________________________________________________________________
dense_9 (Dense)                 (None, 324)          105300      flatten_1[0][0]                  
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 45)           23085       leaky_re_lu_3[0][0]              
__________________________________________________________________________________________________
leaky_re_lu_9 (LeakyReLU)       (None, 324)          0           dense_9[0][0]                    
__________________________________________________________________________________________________
leaky_re_lu_4 (LeakyReLU)       (None, 45)           0           dense_4[0][0]                    
__________________________________________________________________________________________________
reshape_3 (Reshape)             (None, 4, 9, 9)      0           leaky_re_lu_9[0][0]              
__________________________________________________________________________________________________
reshape_1 (Reshape)             (None, 9, 5)         0           leaky_re_lu_4[0][0]              
__________________________________________________________________________________________________
lambda_2 (Lambda)               (None, 9, 9, 4)      0           reshape_3[0][0]                  
__________________________________________________________________________________________________
output_1 (Lambda)               (None, 9, 5)         0           reshape_1[0][0]                  
__________________________________________________________________________________________________
output_2 (Lambda)               (None, 9, 9, 4)      0           lambda_2[0][0]                   
==================================================================================================
Total params: 769,437
Trainable params: 769,437
Non-trainable params: 0
__________________________________________________________________________________________________

If you think the model has an issue, please check “model”. This question is different from this question which uses only one output in the loss. Here is also the loss function from the original Tensorflow code:

# -- loss function
Y_1 = tf.placeholder(tf.float32, shape=[None, 9, 9, 4])
Y_2 = tf.placeholder(tf.float32, shape=[None, 9, 5])

loss_1 = tf.nn.softmax_cross_entropy_with_logits(labels=Y_2, logits=fcn(X)[0])
loss_2 = tf.nn.softmax_cross_entropy_with_logits(labels=Y_1, logits=fcn(X)[1])
loss_2 = tf.matrix_band_part(loss_2, 0, -1) - tf.matrix_band_part(loss_2, 0, 0)

loss = tf.reduce_mean(loss_1) + 2 * tf.reduce_mean(loss_2)


Get this bounty!!!

#StackBounty: #python #tensorflow How to utilize 100% of GPU memory with Tensorflow?

Bounty: 50

I have a 32Gb graphics card and upon start of my script I see:

2019-07-11 01:26:19.985367: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 95.16G (102174818304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:19.988090: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 85.64G (91957338112 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:19.990806: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 77.08G (82761605120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:19.993527: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 69.37G (74485440512 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:19.996219: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 62.43G (67036893184 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:19.998911: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 56.19G (60333203456 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:20.001601: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 50.57G (54299881472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:20.004296: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 45.51G (48869892096 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:20.006981: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 40.96G (43982901248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:20.009660: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 36.87G (39584608256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2019-07-11 01:26:20.012341: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 33.18G (35626147840 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

After which TF settles with using 96% of my memory. And later, when it runs out of memory it tryies to allocate 65G

tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 65.30G (70111285248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

My question is, what about remaining 1300MB (0.04*32480)? I would not mind using those before running OOM.

How can I make TF utilize 99.9% of memory instead of 96%?

Update: nvidia-smi output

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.40.04    Driver Version: 418.40.04    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:16.0 Off |                    0 |
| N/A   66C    P0   293W / 300W |  31274MiB / 32480MiB |    100%      Default |

I am asking about these 1205MB (31274MiB – 32480MiB) remaining unused. Maybe they are there for a reason, maybe they are used just before OOM.


Get this bounty!!!

#StackBounty: #python #tensorflow #conv-neural-network #generative-adversarial-network #niftynet Changing CNN to work with 3D convoluti…

Bounty: 100

I am working with code from here (paper here) that creates a GAN. I am trying to apply this to a new domain, switching from their application on MNIST to 3D brain MRI images. My issue is in the defining of the GAN itself.

For example, their code for defining the generative model (takes noise of dimension z_dim and produces an image from the MNIST distribution, so 28×28) is this, with my comments based on how I believe it works:

def generate(self, z):
    # start with noise in compact space
    assert z.shape[1] == self.z_dim

    # Fully connected layer that for some reason expands to latent * 64
    output = tflib.ops.linear.Linear('Generator.Input', self.z_dim,
                                     self.latent_dim * 64, z)
    output = tf.nn.relu(output)
    # Reshape the latent dimension into 4x4 MNIST
    output = tf.reshape(output, [-1, self.latent_dim * 4, 4, 4])

    # Reduce the latent dimension to get 8x8 MNIST
    output = tflib.ops.deconv2d.Deconv2D('Generator.2', self.latent_dim * 4,
                                         self.latent_dim * 2, 5, output)
    output = tf.nn.relu(output)  # 8 x 8
    # To be able to get 28x28 later?
    output = output[:, :, :7, :7]  # 7 x 7

    # Reduce more to get 14x14
    output = tflib.ops.deconv2d.Deconv2D('Generator.3', self.latent_dim * 2,
                                         self.latent_dim, 5, output)
    output = tf.nn.relu(output)  # 14 x 14

    output = tflib.ops.deconv2d.Deconv2D('Generator.Output',
                                         self.latent_dim, 1, 5, output)
    output = tf.nn.sigmoid(output)  # 28 x 28

    if self.gen_params is None:
        self.gen_params = tflib.params_with_name('Generator')

    return tf.reshape(output, [-1, self.x_dim])

And this is my code using niftynet convolutional layers, where z_dim and latent_dim are the same as before at 64, and I’ve added the results of the print statements:

def generate(self, z):
    assert z.shape[1] == self.z_dim

    generator_input = FullyConnectedLayer(self.latent_dim * 64,
                acti_func='relu',
                #with_bn = True,
                name='Generator.Input')
    output = generator_input(z, is_training=True)

    print(output.shape) # (?, 4096)
    #output = tflib.ops.linear.Linear('Generator.Input', self.z_dim,
    #                                 self.latent_dim * 64, z)
    #output = tf.nn.relu(output)
    output = tf.reshape(output, [-1, self.latent_dim * 4, 1, 18, 18])  # 4 x 4

    print(output.shape) # (?, 256, 1, 18, 18)

    generator_2 = DeconvolutionalLayer(self.latent_dim*2,
                    kernel_size=5,
                    stride=2,
                    acti_func='relu',
                    name='Generator.2')
    output = generator_2(output, is_training=True)
    #output = tflib.ops.deconv2d.Deconv2D('Generator.2', self.latent_dim * 4,
    #                                     self.latent_dim * 2, 5, output)
    #output = tf.nn.relu(output)  # 8 x 8
    print(output.shape) # (?, 512, 2, 36, 128)
    #output = output[:, :, :-1, :-1]  # 7 x 7

    generator_3 = DeconvolutionalLayer(self.latent_dim,
                    kernel_size=5,
                    stride=2,
                    acti_func='relu',
                    name='Generator.3')
    output = generator_3(output, is_training=True)
    #output = tflib.ops.deconv2d.Deconv2D('Generator.3', self.latent_dim * 2,
    #                                     self.latent_dim, 5, output)
    #output = tf.nn.relu(output)  # 14 x 14

    print(output.shape) # (?, 1024, 4, 72, 64)

    generator_out = DeconvolutionalLayer(1,
                    kernel_size=5,
                    stride=2,
                    acti_func='sigmoid',
                    name='Generator.Output')
    output = generator_out(output, is_training=True)

    #output = tflib.ops.deconv2d.Deconv2D('Generator.Output',
    #                                     self.latent_dim, 1, 5, output)
    #output = tf.nn.sigmoid(output)  # 28 x 28

    if self.gen_params is None:
        self.gen_params = tflib.params_with_name('Generator')

    print(output.shape) # (?, 2048, 8, 144, 1)
    print("Should be %s" % str(self.x_dim)) # [1, 19, 144, 144, 4]

    return tf.reshape(output, self.x_dim)

I am not really sure how to be able to get the 19 in there. Currently I get this error.

ValueError: Dimension size must be evenly divisible by 2359296 but is 1575936 for ‘Reshape_1’ (op: ‘Reshape’) with input shapes: [?,2048,8,144,1], [5] and with input tensors computed as partial shapes: input1 = [1,19,144,144,4].

I am also relatively new to building NNs and I also have a few questions. What is the point of the latent space when we already have a compact representation in z-space? How do I decide the size of the “output dimension” i.e. the second parameter in the layer constructor? Thank you!


Get this bounty!!!

#StackBounty: #neural-networks #tensorflow #keras #residual-networks #inverse-problem How can using ResNet for concept of nonlinear dyn…

Bounty: 50

To give some background, recently people have realized that specific deep neural networks (DNNs) (such as ResNets and RNN) can be modeled as nonlinear dynamical systems. This opens up the possibility of applying known results from systems theory to the study of neural networks.

Turnpikes:

Something quite interesting is that the training of a DNN can be expressed as an open-loop optimal control problem. I’m interested in seeing if neural networks have “Turnpikes”.

This is a phenomenon that occurs in some optimal control problems where all optimal solutions run “close” to some trajectory, obtained from a static optimization problem. (“Turnpike” is an American term for a “Highway”/”Autobahn”). What’s nice is that if you have the turnpike, then you have a perfect initial guess for the solvers, and you could speed up the training of DNNs.

I studied a paper which investigates on this issue by using MNIST dataset. In this paper, it seems that they observe the behavior of weights as Turnpike! But I’m not sure.

My questions:

  • In this concept, I’m doubtful that whether Weights are Turnpikes or not? if so, Does the goal/case is understanding ResNet Weight Initialization like here?
  • I was wondering if I train some (very deep) ResNets and look at the hidden layers to see if there are turnpikes.
  • Which types of ResNet will be fit for this training: ResNet-32 or ResNet-56 , ResNet-110 ?

Any help would be much appreciated


Get this bounty!!!