linear_remainder
Bases: remainder
The linear remainder function.
It calculates the linear remainder, and returns remainder term of certain length. The identity remainder function can only be applied when the input and output dimensions are equal. For more general cases where the input and output dimensions are different, the linear remainder function can be used. This class inherits from the base remainder class (i.e., the remainder class in the module directory).
...
Notes
The linear remainder function \(\pi: {R}^m \to {R}^n\) just projects all inputs to themselves subject to linear transformations, i.e., $$ \begin{equation} \pi(\mathbf{x}) = \mathbf{x} \mathbf{W} \in {R}^n, \end{equation} $$ where the learnable parameter matrix \(\mathbf{W} \in R^{m \times n}\) is used for vector dimension adjustment.
By default, the remainder term will also be processed with the optional activation functions.
Attributes:
Name | Type | Description |
---|---|---|
name |
str, default = 'linear_remainder'
|
Name of the remainder function. |
Methods:
Name | Description |
---|---|
__init__ |
It initializes the linear remainder function. |
forward |
It implements the abstract forward method declared in the base remainder class. |
Source code in tinybig/remainder/basic_remainder.py
292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 |
|
__init__(name='linear_remainder', require_parameters=True, enable_bias=False, *args, **kwargs)
The initialization method of the linear remainder function.
It initializes a linear remainder function object. This method will also call the initialization method of the base class as well.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name |
str
|
Name of the linear remainder function. |
'linear_remainder'
|
require_parameters |
bool
|
Boolean tag of whether the function requires parameters. |
True
|
enable_bias |
bool
|
Boolean tag of whether the bias is enabled or not. |
False
|
Returns:
Type | Description |
---|---|
object
|
The linear remainder function object. |
Source code in tinybig/remainder/basic_remainder.py
forward(x, w, b=None, device='cpu', *args, **kwargs)
The forward method of the linear remainder function.
The linear remainder function \(\pi: {R}^m \to {R}^n\) just projects all inputs to themselves subject to linear transformations, i.e., $$ \begin{equation} \pi(\mathbf{x}) = \mathbf{x} \mathbf{W} \in {R}^n, \end{equation} $$ where the learnable parameter matrix \(\mathbf{W} \in R^{m \times n}\) is used for vector dimension adjustment.
By default, the remainder term will also be processed with the optional activation functions.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
x |
Tensor
|
The input data vector. |
required |
w |
Parameter
|
The linear transformation parameter. |
required |
b |
Parameter
|
The linear transformation bias parameter. It will not be None if attribute "enable_bias" is assigned with "True" value at initialization. |
None
|
device |
Device to calculate the remainder function. |
'cpu'
|
Returns:
Type | Description |
---|---|
Tensor
|
The remainder term of the input. |