lphm_parameterized_interdependence
Bases: parameterized_interdependence
A parameterized interdependence function using low-rank hierarchical mapping (LPHM).
Notes
Formally, given the parameter vector \(\mathbf{w} \in R^{l_{\xi}}\) and rank hyper-parameter \(r\), we partition \(\mathbf{w}\) into three sub-vectors and subsequently reshape them into three matrices \(\mathbf{A} \in R^{p \times q}\), \(\mathbf{S} \in R^{\frac{m}{p} \times r}\) and \(\mathbf{T} \in R^{\frac{m'}{q} \times r}\).
These three sub-matrices \(\mathbf{A}\), \(\mathbf{S}\) and \(\mathbf{T}\) help define the low-rank hypercomplex parameterized interdependence function as follows:
$$ \begin{equation} \xi(\mathbf{w}) = \mathbf{A} \otimes \mathbf{B} = \mathbf{A} \otimes ( \mathbf{S} \mathbf{T}^\top) \in R^{m \times m'}, \end{equation} $$ whose required length of vector \(\mathbf{w}\) is \(l_{\xi} = pq + r \times (\frac{m}{p} + \frac{m'}{q})\).
Attributes:
Name | Type | Description |
---|---|---|
r |
int
|
Rank of the low-rank approximation. |
p |
int
|
Number of partitions in the input dimension. |
q |
int
|
Number of partitions in the output dimension. |
Methods:
Name | Description |
---|---|
__init__ |
Initializes the low-rank hierarchical mapping parameterized interdependence function. |
Source code in tinybig/interdependence/parameterized_interdependence.py
__init__(r, p, q=None, name='lphm_parameterized_interdependence', *args, **kwargs)
Initializes the low-rank hierarchical mapping (LPHM) parameterized interdependence function.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
r
|
int
|
Rank of the low-rank approximation. |
required |
p
|
int
|
Number of partitions in the input dimension. |
required |
q
|
int
|
Number of partitions in the output dimension. Defaults to |
None
|
name
|
str
|
Name of the interdependence function. Defaults to 'lphm_parameterized_interdependence'. |
'lphm_parameterized_interdependence'
|
*args
|
tuple
|
Additional positional arguments for the parent class. |
()
|
**kwargs
|
dict
|
Additional keyword arguments for the parent class. |
{}
|
Raises:
Type | Description |
---|---|
ValueError
|
If the interdependence type is not supported. |
AssertionError
|
If the input and output dimensions are not divisible by their respective partitions. |