constant_eye_reconciliation
Bases: fabrication
The constant eye parameter reconciliation function.
It performs the constant eye parameter reconciliation, and returns the constant eye parameter matrix of shape (n, D). It is a special case of the constant_reconciliation function defined above. This class inherits from the reconciliation class (i.e., the fabrication class in the module directory).
...
Notes
As a special case of the constant_reconciliation function defined above, the constant eye parameter reconciliation projects any input parameters to constant eye matrix as follows: $$ \begin{equation} \psi(\mathbf{w}) = \mathbf{I}^{n \times D} \in {R}^{n \times D}, \end{equation} $$ where the output matrix \(\mathbf{I}\) of size \(n \times D\) is returned as an eye matrix.
For constant eye parameter reconciliation, the input parameter \(\mathbf{w}\) is not required, which together with its dimension hyper-parameter \(l\) can both be set to \textit{none} in implementation.
Similar as the above constant parameter reconciliation function, the constant eye reconciliation contributes almost nothing to model learning since it involves no learnable parameters, but it provides RPN with substantial flexibility in representing and designing many models.
Attributes:
Name | Type | Description |
---|---|---|
name |
str, default = 'constant_eye_reconciliation'
|
Name of the reconciliation function |
Methods:
Name | Description |
---|---|
__init__ |
It initializes the parameter reconciliation function. |
calculate_l |
It calculates the length of required parameters. |
forward |
It implements the abstract forward method declared in the base reconciliation class. |
Source code in tinybig/reconciliation/basic_reconciliation.py
231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 |
|
__init__(name='constant_eye_reconciliation', *args, **kwargs)
The initialization method of the constant eye parameter reconciliation function.
It initializes a constant eye parameter reconciliation function object. This method will also call the initialization method of the base class as well. Since the constant eye parameter reconciliation doesn't require parameters, it will set the "require_parameters" as False in the initialization.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name |
Name of the constant eye parameter reconciliation function. |
'constant_eye_reconciliation'
|
Returns:
Type | Description |
---|---|
object
|
The constant eye parameter reconciliation function object. |
Source code in tinybig/reconciliation/basic_reconciliation.py
calculate_l(n, D)
The required parameter number calculation method.
It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function based on the intermediate and output space dimensions, n and D. For constant eye parameter reconciliation, it doesn't require any learnable parameters, and this function will return the parameter number as 0 by default.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
n |
int
|
The dimension of the output space. |
required |
D |
int
|
The dimension of the intermediate expansion space. |
required |
Returns:
Type | Description |
---|---|
int
|
The number of required learnable parameters. |
Source code in tinybig/reconciliation/basic_reconciliation.py
forward(n, D, w=None, device='cpu', *args, **kwargs)
The forward method of the parameter reconciliation function.
It applies the constant eye parameter reconciliation operation to the input parameter of length l, and returns the reconciled parameter matrix of shape (n, D) as follows: $$ \begin{equation} \psi(\mathbf{w}) = \mathbf{I}^{n \times D} \in {R}^{n \times D}, \end{equation} $$ where the output matrix \(\mathbf{I}\) of size \(n \times D\) is returned as an eye matrix.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
n |
int
|
The dimension of the output space. |
required |
D |
int
|
The dimension of the intermediate expansion space. |
required |
w |
Parameter
|
The learnable parameters of the model. For constant eye reconciliation, it is assigned with a default value None. |
None
|
device |
Device to perform the parameter reconciliation. |
'cpu'
|
Returns:
Type | Description |
---|---|
Tensor
|
The reconciled parameter matrix of shape (n, D). |