masking_reconciliation
Bases: fabrication
The masking parameter reconciliation function.
It performs the masking parameter reconciliation, and returns the masked parameter matrix of shape (n, D). This class inherits from the reconciliation class (i.e., the fabrication class in the module directory).
...
Notes
To mitigate the identified limitation of identity parameter reconciliation function, the masking parameter reconciliation function curtail the count of learnable parameters in \(\mathbf{W}\) to a reduced number of \(l\) via a randomly generated masking matrix \(\mathbf{M}\) as follows: $$ \begin{equation} \psi({\mathbf{w}}) = (\mathbf{M} \odot \text{reshape}(\mathbf{w})) = (\mathbf{M} \odot \mathbf{W}) \in {R}^{n \times D}, \end{equation} $$ where the term \(\mathbf{M} \in \{0, 1\}^{n \times D}\) denotes the binary masking matrix only with \(l\) non-zero entries and \(\odot\) denotes the element-wise product operator.
To facilitate practical implementation, instead of pre-define the parameter dimension \(l\), the masking reconciliation function uses the masking ratio \(p\) as a parameter of the masking based reconciliation function instead. This parameter, in conjunction with the output dimensions \(n \times D\), computes the requisite parameter vector dimension, shown as follows: $$ \begin{equation} l = p \times n \times D, \end{equation} $$ where the masking ratio takes value from \(p \in [0, 1]\). For masking_ratio p=1.0: all parameters are used; while masking_ratio p=0.0: no parameters will be used.
Attributes:
Name | Type | Description |
---|---|---|
name |
str, default = 'masking_reconciliation'
|
Name of the parameter reconciliation function |
p |
float, default = 0.5
|
The masking ratio of elements in the parameter matrix, which denotes the percentage of used parameter, e.g., masking_ratio p=1.0: all parameters are used; masking_ratio p=0.0: no parameters will be used. |
fixed_mask |
bool, default = True
|
Whether the masking matrix is fixed for all inputs or not. |
Methods:
Name | Description |
---|---|
__init__ |
It initializes the parameter reconciliation function. |
calculate_l |
It calculates the length of required parameters. |
forward |
It implements the abstract forward method declared in the base reconciliation class. |
Source code in tinybig/reconciliation/basic_reconciliation.py
478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 |
|
__init__(name='masking_reconciliation', p=0.5, fixed_mask=True, *args, **kwargs)
The initialization method of the masking parameter reconciliation function.
It initializes a masking parameter reconciliation function object. This method will also call the initialization method of the base class as well.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name |
Name of the parameter reconciliation function. |
'masking_reconciliation'
|
|
p |
The masking ratio of elements in the parameter matrix, which denotes the percentage of used parameter, e.g., masking_ratio p=1.0: all parameters are used; masking_ratio p=0.0: no parameters will be used. |
0.5
|
|
fixed_mask |
bool
|
Whether the masking matrix is fixed for all inputs or not. |
True
|
Returns:
Type | Description |
---|---|
object
|
The masking parameter reconciliation function object. |
Source code in tinybig/reconciliation/basic_reconciliation.py
calculate_l(n, D)
The required parameter number calculation method.
It calculates the number of required learnable parameters, i.e., \(l\), of the parameter reconciliation function based on the intermediate and output space dimensions, \(n\) and \(D\), and masking ratio parameter \(p\), which can be represented as follows: $$ \begin{equation} l = p \times n \times D. \end{equation} $$
Parameters:
Name | Type | Description | Default |
---|---|---|---|
n |
int
|
The dimension of the output space. |
required |
D |
int
|
The dimension of the intermediate expansion space. |
required |
Returns:
Type | Description |
---|---|
int
|
The number of required learnable parameters. |
Source code in tinybig/reconciliation/basic_reconciliation.py
forward(n, D, w, device='cpu', *args, **kwargs)
The forward method of the parameter reconciliation function.
It applies the masking parameter reconciliation operation to the input parameter vector, and returns the reconciled parameter matrix of shape (n, D) subject to the masking ratio \(p\) as follows: $$ \begin{equation} \psi({\mathbf{w}}) = (\mathbf{M} \odot \text{reshape}(\mathbf{w})) = (\mathbf{M} \odot \mathbf{W}) \in {R}^{n \times D}, \end{equation} $$
Parameters:
Name | Type | Description | Default |
---|---|---|---|
n |
int
|
The dimension of the output space. |
required |
D |
int
|
The dimension of the intermediate expansion space. |
required |
w |
Parameter
|
The learnable parameters of the model. |
required |
device |
Device to perform the parameter reconciliation. |
'cpu'
|
Returns:
Type | Description |
---|---|
Tensor
|
The reconciled parameter matrix of shape (n, D). |
Source code in tinybig/reconciliation/basic_reconciliation.py
generate_masking_matrix(n, D)
The masking matrix generation method.
It generates the masking matrix of shape (n, D) subject to the masking ratio parameter \(p\). The method first randomly generates a matrix of shape (n, D) and then compares it with parameter \(p\) to define the binary masking matrix.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
n |
The dimension of the output space. |
required | |
D |
The dimension of the intermediate expansion space. |
required |
Returns:
Type | Description |
---|---|
Tensor
|
The binary masking matrix of shape (n, D). |