Hyperparameters

Hyperparameters are set before the learning process begins, and they affect how the model is trained and performs.

Here are the key hyperparameters that you can set:

1. Optimizer

An optimizer is a crucial component used during the training of machine learning models, particularly neural networks. The optimizer's primary function is to update the model's parameters (e.g., weights and biases) in a way that minimizes a specified loss or cost function. It takes a string as its value, which specifies the name of the optimizer to use. The default optimizer is SGD for both PyTorch and tensorflow.

TensorFlow
This is similar to the built-in optimizer function in TensorFlow. Some of the supported optimizers include Adam, RMSprop, SGD, Adadelta, Adagrad, Adamax, Nadam, and FTRL.
PyTorch
This is similar to the built-in optimizer function. Some of the supported optimizers include Adam, RMSprop, SGD, Adadelta, Adagrad and Adamax.

To set an optimizer, you can use the following command:

trainingObject.optimizer('rmsprop')

2. Learning rate

Learning Rate controls the rate at which the model learns. Specifically, it controls the amount of error that the weights of the model are updated with each time they are updated (e.g. at the end of each batch of training examples). It takes a dictionary/json as its value, which specifies the type of learning rate you want to use and the value of the learning or the name of learning rate function.

The default value is {'type': 'constant', 'value': 0.001}.

We support three different types of learning rates:

Constant: To set the learning rate to constant, use the following command:
trainingObject.learningRate({'type': 'constant', 'value': 0.002})
Adaptive: To use an adaptive learning rate, pass the dictionary containing the different parameters and their values as key value pairs:
trainingObject.learningRateAdaptive({'type': 'adaptive', 'schedular': 'ExponentialDecay', 'decay_steps':1000, 'decay_rate':0.5})

Custom: To set a custom learning rate, use a custom learning rate function like the one below. Then call the method learningRateCustom method of trainingObject and pass the corresponding values:

```python
def custom_LearningRate_schedular(epoch):
        if epoch < 5:
            return 0.01
        else:
            return 0.01 * tf.math.exp(0.1 * (10 - epoch))
trainingObject.learningRateCustom({'type': 'constant', 'value': custom_LearningRate_schedular, 'epoch': 4})
```

TensorFlow: All three types of learning rates are supported. PyTorch: Only constant learning rates are supported.

3. Loss function

This is similar to the built-in loss function in TensorFlow. It takes a dictionary as its value, which specifies the type of loss function and the name of the loss function to use as its value. Some of the supported loss functions include binary_crossentropy, categorical_crossentropy, mse for TensorFlow and crossentropy, mse, l1 for PyTorch.

The default loss function is {'type': 'standard', 'value': 'categorical_crossentropy'}.

We support two different types of loss functions:

Standard: To set a standard loss function, you can use the following command:

trainingObject.lossFunction({'type': 'standard', 'value': 'categorical_crossentropy'})

Custom: To set a custom loss function, implement your custom loss function as a function, for example like the one below. Then call the method lossFunctionCustom method of trainingObject and pass the corresponding values:

```python
def custom_mse(y_true, y_pred):
    # calculating squared difference between target and predicted values
    loss = K.square(y_pred - y_true)  # (batch_size, 2)

    # multiplying the values with weights along batch dimension
    loss = loss * [0.3, 0.7]          # (batch_size, 2)

    # summing both loss values along batch dimension
    loss = K.sum(loss, axis=1)        # (batch_size,)

    return loss
trainingObject.lossFunctionCustom({'type': 'custom', 'value':custom_mse})
```

TensorFlow: Both standard and custom loss functions are supported. PyTorch: Only the standard loss functions are supported.

4. Layers freeze

This parameter is used to specify which layers of the model should be "frozen" (i.e. not updated) while the model is training. If you specify the names of layers that are not present in the model, or if you provide incorrect names, then all layers will be trained by default.

To set which layers should be frozen, you can use the following command:

trainingObject.layersFreeze(['conv1','fc1'])