Skip to content

remainder

Bases: Module, function

The base class of the remainder function in the tinyBIG toolkit.

It will be used as the base class template for defining the remainder functions.

...

Notes

Formally, to approximate the underlying data distribution mapping \(f: {R}^m \to {R}^n\) to be learned, in addition to the data expansion function and parameter reconciliation function, the remainder function \(\pi\) completes the approximation as a residual term, governing the learning completeness of the RPN model, which can be represented as follows

\[ \pi: {R}^m \to {R}^{n}.\]

Without specific descriptions, the remainder function \(\pi\) defined here is based solely on the input data \(\mathbf{x}\). However, in practice, we also allow \(\pi\) to include learnable parameters for output dimension adjustment. In such cases, it should be rewritten as \(\pi(\mathbf{x} | \mathbf{w}')\), where \(\mathbf{w}'\) is one extra fraction of the model's learnable parameters.

Attributes:

Name Type Description
name str, default = 'base_remainder'

Name of the remainder function.
require_remainder_parameters bool, default = False

Boolean tag of whether the function requires parameters.
enable_remainder_bias bool, default = False

Boolean tag of whether the bias is enabled or not.
activation_functions list, default = None

The list of activation functions that can be applied in the remainder function.
activation_function_configs list, default = None

The list of activation function configs that can be applied in the remainder function.
device str, default = 'cpu'

Device of the remainder function.

Methods:

Name Description
__init__

It initializes the remainder function.
get_name

It gets the name of the remainder function.
activation

It applies the activation functions to data calculated in this remainder function.
forward

The forward method to calculate the remainder term.
__call__

The build-in callable method of the remainder function.
Source code in tinybig/module/base_remainder.py 
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
class remainder(Module, function):
    r"""
    The base class of the remainder function in the tinyBIG toolkit.

    It will be used as the base class template for defining the remainder functions.

    ...

    Notes
    ----------
    Formally, to approximate the underlying data distribution mapping $f: {R}^m \to {R}^n$ to be learned,
    in addition to the data expansion function and parameter reconciliation function, the remainder function
    $\pi$ completes the approximation as a residual term, governing the learning completeness of the RPN model,
    which can be represented as follows

    $$ \pi: {R}^m \to {R}^{n}.$$

    Without specific descriptions, the remainder function $\pi$ defined here is based solely on the input data $\mathbf{x}$.
    However, in practice, we also allow $\pi$ to include learnable parameters for output dimension adjustment.
    In such cases, it should be rewritten as $\pi(\mathbf{x} | \mathbf{w}')$, where $\mathbf{w}'$ is one extra fraction of the
    model's learnable parameters.

    Attributes
    ----------
    name: str, default = 'base_remainder'
        Name of the remainder function.
    require_remainder_parameters: bool, default = False
        Boolean tag of whether the function requires parameters.
    enable_remainder_bias: bool, default = False
        Boolean tag of whether the bias is enabled or not.
    activation_functions: list, default = None
        The list of activation functions that can be applied in the remainder function.
    activation_function_configs: list, default = None
        The list of activation function configs that can be applied in the remainder function.
    device: str, default = 'cpu'
        Device of the remainder function.

    Methods
    ----------
    __init__
        It initializes the remainder function.

    get_name
        It gets the name of the remainder function.

    activation
        It applies the activation functions to data calculated in this remainder function.

    forward
        The forward method to calculate the remainder term.

    __call__
        The build-in callable method of the remainder function.
    """
    def __init__(
        self,
        name='base_remainder',
        require_parameters=False,
        enable_bias=False,
        activation_functions=None,
        activation_function_configs=None,
        device='cpu',
        *args, **kwargs
    ):
        """
        The initialization method of the base remainder function.

        It initializes a base remainder function object.

        Parameters
        ----------
        name: str, default = 'base_remainder'
            Name of the remainder function.
        require_parameters: bool, default = False
            Boolean tag of whether the function requires parameters.
        enable_bias: bool, default = False
            Boolean tag of whether the bias is enabled or not.
        activation_functions: list, default = None
            The list of activation functions that can be applied in the remainder function.
        activation_function_configs: list, default = None
            The list of activation function configs that can be applied in the remainder function.
        device: str, default = 'cpu'
            Device of the remainder function.

        Returns
        ----------
        object
            The remainder function object.
        """
        Module.__init__(self)
        function.__init__(self, name=name, device=device)

        self.require_parameters = require_parameters
        self.enable_bias = enable_bias
        self.activation_functions = config.instantiation_functions(activation_functions, activation_function_configs, device=self.device)

    def get_name(self):
        """
        The name retrieval method of the remainder function.

        It returns the name of the remainder function.

        Returns
        -------
        str
            The name of the remainder function.
        """

        return self.name

    def activation(self, x: torch.Tensor, device='cpu', *args, **kwargs):
        """
        The activation method of remainder function.

        It processes the remainder term with the (optional) activation functions.

        Parameters
        ----------
        x: torch.Tensor
            The input data vector.
        device: str, default = 'cpu'
            The device to perform the data expansion.

        Returns
        -------
        Tensor
            It returns the updated remainder term processed by the activation functions.
        """
        return function.func_x(x, self.activation_functions, device=device)

    def to_config(self):
        class_name = f"{self.__class__.__module__}.{self.__class__.__name__}"
        attributes = {attr: getattr(self, attr) for attr in self.__dict__}
        attributes.pop('activation_functions')

        if self.activation_functions is not None:
            attributes['activation_function_configs'] = function.functions_to_configs(self.activation_functions)

        return {
            "function_class": class_name,
            "function_parameters": attributes
        }

    def __call__(self, *args, **kwargs):
        """
        The re-implementation of the callable method.

        It calculates the remainder term based on the inputs. For some remainder functions, this method
        will also accept parameters as the input, which can be applied to the input data for remainder calculation.
        This method will execute by calling the "forward" method.

        Returns
        ----------
        torch.Tensor
            The remainder term calculated based on the input.
        """
        return self.forward(*args, **kwargs)

    @abstractmethod
    def forward(self, *args, **kwargs):
        """
        The forward method of the remainder function.

        It calculates the remainder term based on the inputs. For some remainder functions, this method
        will also accept parameters as the input, which can be applied to the input data for remainder calculation.
        The method is declared as an abstractmethod and needs to be implemented in the inherited classes.

        Returns
        ----------
        torch.Tensor
            The remainder term calculated based on the input.
        """
        pass

__call__(*args, **kwargs)

The re-implementation of the callable method.

It calculates the remainder term based on the inputs. For some remainder functions, this method will also accept parameters as the input, which can be applied to the input data for remainder calculation. This method will execute by calling the "forward" method.

Returns:

Type Description
Tensor

The remainder term calculated based on the input.
Source code in tinybig/module/base_remainder.py 
168
169
170
171
172
173
174
175
176
177
178
179
180
181
def __call__(self, *args, **kwargs):
    """
    The re-implementation of the callable method.

    It calculates the remainder term based on the inputs. For some remainder functions, this method
    will also accept parameters as the input, which can be applied to the input data for remainder calculation.
    This method will execute by calling the "forward" method.

    Returns
    ----------
    torch.Tensor
        The remainder term calculated based on the input.
    """
    return self.forward(*args, **kwargs)

__init__(name='base_remainder', require_parameters=False, enable_bias=False, activation_functions=None, activation_function_configs=None, device='cpu', *args, **kwargs)

The initialization method of the base remainder function.

It initializes a base remainder function object.

Parameters:

Name Type Description Default
name

Name of the remainder function.

 'base_remainder'
require_parameters

Boolean tag of whether the function requires parameters.

 False
enable_bias

Boolean tag of whether the bias is enabled or not.

 False
activation_functions

The list of activation functions that can be applied in the remainder function.

 None
activation_function_configs

The list of activation function configs that can be applied in the remainder function.

 None
device

Device of the remainder function.

 'cpu'

Returns:

Type Description
object

The remainder function object.
Source code in tinybig/module/base_remainder.py 
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
def __init__(
    self,
    name='base_remainder',
    require_parameters=False,
    enable_bias=False,
    activation_functions=None,
    activation_function_configs=None,
    device='cpu',
    *args, **kwargs
):
    """
    The initialization method of the base remainder function.

    It initializes a base remainder function object.

    Parameters
    ----------
    name: str, default = 'base_remainder'
        Name of the remainder function.
    require_parameters: bool, default = False
        Boolean tag of whether the function requires parameters.
    enable_bias: bool, default = False
        Boolean tag of whether the bias is enabled or not.
    activation_functions: list, default = None
        The list of activation functions that can be applied in the remainder function.
    activation_function_configs: list, default = None
        The list of activation function configs that can be applied in the remainder function.
    device: str, default = 'cpu'
        Device of the remainder function.

    Returns
    ----------
    object
        The remainder function object.
    """
    Module.__init__(self)
    function.__init__(self, name=name, device=device)

    self.require_parameters = require_parameters
    self.enable_bias = enable_bias
    self.activation_functions = config.instantiation_functions(activation_functions, activation_function_configs, device=self.device)

activation(x, device='cpu', *args, **kwargs)

The activation method of remainder function.

It processes the remainder term with the (optional) activation functions.

Parameters:

Name Type Description Default
x Tensor

The input data vector.

 required
device

The device to perform the data expansion.

 'cpu'

Returns:

Type Description
Tensor

It returns the updated remainder term processed by the activation functions.
Source code in tinybig/module/base_remainder.py 
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
def activation(self, x: torch.Tensor, device='cpu', *args, **kwargs):
    """
    The activation method of remainder function.

    It processes the remainder term with the (optional) activation functions.

    Parameters
    ----------
    x: torch.Tensor
        The input data vector.
    device: str, default = 'cpu'
        The device to perform the data expansion.

    Returns
    -------
    Tensor
        It returns the updated remainder term processed by the activation functions.
    """
    return function.func_x(x, self.activation_functions, device=device)

forward(*args, **kwargs) abstractmethod

The forward method of the remainder function.

It calculates the remainder term based on the inputs. For some remainder functions, this method will also accept parameters as the input, which can be applied to the input data for remainder calculation. The method is declared as an abstractmethod and needs to be implemented in the inherited classes.

Returns:

Type Description
Tensor

The remainder term calculated based on the input.
Source code in tinybig/module/base_remainder.py 
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
@abstractmethod
def forward(self, *args, **kwargs):
    """
    The forward method of the remainder function.

    It calculates the remainder term based on the inputs. For some remainder functions, this method
    will also accept parameters as the input, which can be applied to the input data for remainder calculation.
    The method is declared as an abstractmethod and needs to be implemented in the inherited classes.

    Returns
    ----------
    torch.Tensor
        The remainder term calculated based on the input.
    """
    pass

get_name()

The name retrieval method of the remainder function.

It returns the name of the remainder function.

Returns:

Type Description
str

The name of the remainder function.
Source code in tinybig/module/base_remainder.py 
121
122
123
124
125
126
127
128
129
130
131
132
133
def get_name(self):
    """
    The name retrieval method of the remainder function.

    It returns the name of the remainder function.

    Returns
    -------
    str
        The name of the remainder function.
    """

    return self.name