dual_lphm_reconciliation
Bases: fabrication
The dual low-rank parameterized hypercomplex multiplication (Dual-LPHM) based parameter reconciliation function.
It performs the Dual-LPHM parameter reconciliation, and returns the Dual-LPHM reconciled parameter matrix of shape (n, D). This class inherits from the reconciliation class (i.e., the fabrication class in the module directory).
The dual low-rank parameterized hypercomplex multiplication based parameter reconciliation can be viewed as a more agreesive version of the LPHM based parameter reconciliation function. It replaces both \(\mathbf{A}\) and \(\mathbf{B}\) in the hypercomplex multiplication based parameter reconciliation with the products of two low-rank sub-matrices, respectively.
...
Notes
Formally, given the parameter vector \(\mathbf{w} \in {R}^{l}\) and a rank hyper-parameter \(r\), together with the parameter sub-matrix dimension parameters \(p\) and \(q\), the Dual-LPHM reconciliation function partitions \(\mathbf{w}\) into four sub-vectors and subsequently reshapes them into three matrices \(\mathbf{P} \in {R}^{p \times r}\), \(\mathbf{Q} \in {R}^{q \times r}\), \(\mathbf{S} \in {R}^{\frac{n}{p} \times r}\) and \(\mathbf{T} \in {R}^{\frac{D}{q} \times r}\). These sub-matrices \(\mathbf{P}\), \(\mathbf{Q}\), \(\mathbf{S}\) and \(\mathbf{T}\) help define the Dual-LPHM reconciliation function as follows: $$ \begin{equation} \psi(\mathbf{w}) = \mathbf{A} \otimes \mathbf{B} = ( \mathbf{P} \mathbf{Q}^\top) \otimes ( \mathbf{S} \mathbf{T}^\top) \in {R}^{n \times D}. \end{equation} $$ This necessitates imposing certain limitations on these dimension and rank parameters, and the parameter vector length \(l\) can be calculated as follows: $$ \begin{equation} l = r( p + q + \frac{n}{p} + \frac{D}{q} ). \end{equation} $$
For the Dual-LPHM parameter reconciliation function, it adds strict constraints on the parameters \(p\) and \(q\), which should be the divisors of the target dimensions \(n\) and \(D\), respectively, i.e., $$ \begin{equation} n \% p = 0 \text{, and } D \% q = 0. \end{equation} $$
Attributes:
Name | Type | Description |
---|---|---|
name |
str, default = 'dual_lphm_reconciliation'
|
Name of the Dual-LPHM parameter reconciliation function |
p |
int, default = 2
|
Parameter sub-matrix row dimension. |
q |
int, default = None
|
Parameter sub-matrix column dimension. If q is not provided with initial values, it will be assigned with value p by default. |
r |
int, default = 2
|
Submatrix rank parameter. |
Methods:
Name | Description |
---|---|
__init__ |
It initializes the Dual-LPHM parameter reconciliation function. |
calculate_l |
It calculates the length of required parameters for the reconciliation function. |
forward |
It implements the abstract forward method declared in the base reconciliation class. |
Source code in tinybig/reconciliation/lowrank_reconciliation.py
480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 |
|
__init__(name='dual_lphm_reconciliation', p=2, q=None, r=2, *args, **kwargs)
The initialization method of the Dual-LPHM parameter reconciliation function.
It initializes a Dual-LPHM parameter reconciliation function object. This method will also call the initialization method of the base class as well.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name |
Name of the Dual-LPHM parameter reconciliation function. |
'dual_lphm_reconciliation'
|
|
p |
Parameter sub-matrix row dimension. |
2
|
|
q |
Parameter sub-matrix column dimension. If q is not provided with initial values, it will be assigned with value p by default. |
None
|
|
r |
Submatrix rank parameter. |
2
|
Returns:
Type | Description |
---|---|
object
|
The Dual-LPHM parameter reconciliation function object. |
Source code in tinybig/reconciliation/lowrank_reconciliation.py
calculate_l(n, D)
The required parameter number calculation method.
It calculates the number of required learnable parameters, i.e., \(l\), of the parameter reconciliation function based on the intermediate and output space dimensions, \(n\) and \(D\), and the dimension and rank parameters \(p\), \(q\) and \(r\), which can be represented as follows: $$ \begin{equation} l = r( p + q + \frac{n}{p} + \frac{D}{q} ). \end{equation} $$
Notes
For the Dual-LPHM parameter reconciliation function, it adds strict constraints on the parameters \(p\) and \(q\), which should be the divisors of the target dimensions \(n\) and \(D\), respectively, i.e., $$ \begin{equation} n \% p = 0 \text{, and } D \% q = 0. \end{equation} $$
Parameters:
Name | Type | Description | Default |
---|---|---|---|
n |
int
|
The dimension of the output space. |
required |
D |
int
|
The dimension of the intermediate expansion space. |
required |
Returns:
Type | Description |
---|---|
int
|
The number of required learnable parameters. |
Source code in tinybig/reconciliation/lowrank_reconciliation.py
forward(n, D, w, device='cpu', *args, **kwargs)
The forward method of the parameter reconciliation function.
It applies the Dual-LPHM parameter reconciliation operation to the input parameter vector \(\mathbf{w}\), and returns the reconciled parameter matrix of shape (n, D) subject to the dimension and rank parameters \(p\), \(q\) and \(r\) as follows: $$ \begin{equation} \psi(\mathbf{w}) = \mathbf{A} \otimes \mathbf{B} = ( \mathbf{P} \mathbf{Q}^\top) \otimes ( \mathbf{S} \mathbf{T}^\top) \in {R}^{n \times D}. \end{equation} $$ where \(\mathbf{P} \in {R}^{p \times r}\), \(\mathbf{Q} \in {R}^{q \times r}\), \(\mathbf{S} \in {R}^{\frac{n}{p} \times r}\) and \(\mathbf{T} \in {R}^{\frac{D}{q} \times r}\) are all obtained by partitioning \(\mathbf{w}\) into sub-vectors and subsequently reshaping them into matrices.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
n |
int
|
The dimension of the output space. |
required |
D |
int
|
The dimension of the intermediate expansion space. |
required |
w |
Parameter
|
The learnable parameters of the model. |
required |
device |
Device to perform the parameter reconciliation. |
'cpu'
|
Returns:
Type | Description |
---|---|
Tensor
|
The reconciled parameter matrix of shape (n, D). |