Skip to content

constant_reconciliation

Bases: fabrication

The constant parameter reconciliation function.

It performs the constant parameter reconciliation, and returns the constant reconciled parameter matrix of shape (n, D). This class inherits from the reconciliation class (i.e., the fabrication class in the module directory).

...

Notes

As the simplest parameter reconciliation function, the constant parameter reconciliation projects any input parameters to constants (e.g., zeros or ones) as follows: $$ \begin{equation} \psi(\mathbf{w} | c) = c \cdot \mathbf{1}^{n \times D} = \mathbf{C} \in {R}^{n \times D}, \end{equation} $$ where the output matrix \(\mathbf{C}\) of size \(n \times D\) is filled with the provided constant \(c\).

For constant parameter reconciliation, the input parameter \(\mathbf{w}\) is not required, which together with its dimension hyper-parameter \(l\) can both be set to \textit{none} in implementation.

If the output constant \(\mathbf{C} = \mathbf{0}\) or \(\mathbf{C} = \mathbf{1}\), we can also name the functions as zero reconciliation and one reconciliation, respectively. Constant parameter reconciliation functions can accommodate outputs according to requirements.

Constant reconciliation contributes almost nothing to model learning since it involves no parameters, but it provides RPN with substantial flexibility in representing and designing many models.

Attributes:

Name Type Description
name str, default = 'constant_reconciliation'

Name of the reconciliation function.
c float, default = 1.0

The constant value of the reconciliation function.

Methods:

Name Description
__init__

It initializes the parameter reconciliation function.
calculate_l

It calculates the length of required parameters.
forward

It implements the abstract forward method declared in the base reconciliation class.
Source code in tinybig/reconciliation/basic_reconciliation.py 
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
class constant_reconciliation(fabrication):
    r"""
    The constant parameter reconciliation function.

    It performs the constant parameter reconciliation, and returns the constant reconciled parameter matrix of shape (n, D).
    This class inherits from the reconciliation class (i.e., the fabrication class in the module directory).

    ...

    Notes
    ----------
    As the simplest parameter reconciliation function, the **constant parameter reconciliation** projects
    any input parameters to constants (e.g., zeros or ones) as follows:
    $$
        \begin{equation}
            \psi(\mathbf{w} | c) = c \cdot \mathbf{1}^{n \times D} = \mathbf{C} \in {R}^{n \times D},
        \end{equation}
    $$
    where the output matrix $\mathbf{C}$ of size $n \times D$ is filled with the provided constant $c$.

    For constant parameter reconciliation, the input parameter $\mathbf{w}$ is not required,
    which together with its dimension hyper-parameter $l$ can both be set to \textit{none} in implementation.

    If the output constant $\mathbf{C} = \mathbf{0}$ or $\mathbf{C} = \mathbf{1}$,
    we can also name the functions as **zero reconciliation** and **one reconciliation**, respectively.
    Constant parameter reconciliation functions can accommodate outputs according to requirements.

    Constant reconciliation contributes almost nothing to model learning since it involves no parameters,
    but it provides RPN with substantial flexibility in representing and designing many models.

    Attributes
    ----------
    name: str, default = 'constant_reconciliation'
        Name of the reconciliation function.
    c: float, default = 1.0
        The constant value of the reconciliation function.

    Methods
    ----------
    __init__
        It initializes the parameter reconciliation function.

    calculate_l
        It calculates the length of required parameters.

    forward
        It implements the abstract forward method declared in the base reconciliation class.
    """
    def __init__(self, name: str = 'constant_c_reconciliation', c: float = 1.0, *args, **kwargs):
        """
        The initialization method of the constant parameter reconciliation function.

        It initializes a constant parameter reconciliation function object.
        This method will also call the initialization method of the base class as well.
        Since the constant parameter reconciliation doesn't require parameters, it will
        set the "require_parameters" as False in the initialization.

        Parameters
        ----------
        name: str, default = 'constant_c_reconciliation'
            Name of the constant parameter reconciliation function.
        c: float, default = 1.0
            The constant value of the reconciliation function

        Returns
        ----------
        fabrication
            The constant parameter reconciliation function object.
        """
        super().__init__(name=name, require_parameters=False, *args, **kwargs)
        self.c = c

    def calculate_l(self, n: int, D: int):
        """
        The required parameter number calculation method.

        It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function
        based on the intermediate and output space dimensions, n and D.
        For constant parameter reconciliation, it doesn't require any learnable parameters, and this function
        will return the parameter number as 0 by default.

        Parameters
        ----------
        n: int
            The dimension of the output space.
        D: int
            The dimension of the intermediate expansion space.

        Returns
        -------
        int
            The number of required learnable parameters.
        """
        return 0

    def forward(self, n: int, D: int, w: torch.nn.Parameter = None, device='cpu', *args, **kwargs):
        r"""
        The forward method of the parameter reconciliation function.

        It applies the constant parameter reconciliation operation to the input parameter of length l,
        and returns the reconciled parameter matrix of shape (n, D) as follows:
        $$
            \begin{equation}
                \psi(\mathbf{w} | c) = c \cdot \mathbf{1}^{n \times D} = \mathbf{C} \in {R}^{n \times D}.
            \end{equation}
        $$

        Parameters
        ----------
        n: int
            The dimension of the output space.
        D: int
            The dimension of the intermediate expansion space.
        w: torch.nn.Parameter, default = None
            The learnable parameters of the model.
            For constant reconciliation, it is assigned with a default value None.
        device: str, default = 'cpu'
            Device to perform the parameter reconciliation.

        Returns
        ----------
        torch.Tensor
            The reconciled parameter matrix of shape (n, D).
        """
        return self.c * torch.ones(n, D, device=device)

__init__(name='constant_c_reconciliation', c=1.0, *args, **kwargs)

The initialization method of the constant parameter reconciliation function.

It initializes a constant parameter reconciliation function object. This method will also call the initialization method of the base class as well. Since the constant parameter reconciliation doesn't require parameters, it will set the "require_parameters" as False in the initialization.

Parameters:

Name Type Description Default
name str

Name of the constant parameter reconciliation function.

 'constant_c_reconciliation'
c float

The constant value of the reconciliation function

 1.0

Returns:

Type Description
fabrication

The constant parameter reconciliation function object.
Source code in tinybig/reconciliation/basic_reconciliation.py 
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
def __init__(self, name: str = 'constant_c_reconciliation', c: float = 1.0, *args, **kwargs):
    """
    The initialization method of the constant parameter reconciliation function.

    It initializes a constant parameter reconciliation function object.
    This method will also call the initialization method of the base class as well.
    Since the constant parameter reconciliation doesn't require parameters, it will
    set the "require_parameters" as False in the initialization.

    Parameters
    ----------
    name: str, default = 'constant_c_reconciliation'
        Name of the constant parameter reconciliation function.
    c: float, default = 1.0
        The constant value of the reconciliation function

    Returns
    ----------
    fabrication
        The constant parameter reconciliation function object.
    """
    super().__init__(name=name, require_parameters=False, *args, **kwargs)
    self.c = c

calculate_l(n, D)

The required parameter number calculation method.

It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function based on the intermediate and output space dimensions, n and D. For constant parameter reconciliation, it doesn't require any learnable parameters, and this function will return the parameter number as 0 by default.

Parameters:

Name Type Description Default
n int

The dimension of the output space.

 required
D int

The dimension of the intermediate expansion space.

 required

Returns:

Type Description
int

The number of required learnable parameters.
Source code in tinybig/reconciliation/basic_reconciliation.py 
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
def calculate_l(self, n: int, D: int):
    """
    The required parameter number calculation method.

    It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function
    based on the intermediate and output space dimensions, n and D.
    For constant parameter reconciliation, it doesn't require any learnable parameters, and this function
    will return the parameter number as 0 by default.

    Parameters
    ----------
    n: int
        The dimension of the output space.
    D: int
        The dimension of the intermediate expansion space.

    Returns
    -------
    int
        The number of required learnable parameters.
    """
    return 0

forward(n, D, w=None, device='cpu', *args, **kwargs)

The forward method of the parameter reconciliation function.

It applies the constant parameter reconciliation operation to the input parameter of length l, and returns the reconciled parameter matrix of shape (n, D) as follows: $$ \begin{equation} \psi(\mathbf{w} | c) = c \cdot \mathbf{1}^{n \times D} = \mathbf{C} \in {R}^{n \times D}. \end{equation} $$

Parameters:

Name Type Description Default
n int

The dimension of the output space.

 required
D int

The dimension of the intermediate expansion space.

 required
w Parameter

The learnable parameters of the model. For constant reconciliation, it is assigned with a default value None.

 None
device

Device to perform the parameter reconciliation.

 'cpu'

Returns:

Type Description
Tensor

The reconciled parameter matrix of shape (n, D).
Source code in tinybig/reconciliation/basic_reconciliation.py 
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
def forward(self, n: int, D: int, w: torch.nn.Parameter = None, device='cpu', *args, **kwargs):
    r"""
    The forward method of the parameter reconciliation function.

    It applies the constant parameter reconciliation operation to the input parameter of length l,
    and returns the reconciled parameter matrix of shape (n, D) as follows:
    $$
        \begin{equation}
            \psi(\mathbf{w} | c) = c \cdot \mathbf{1}^{n \times D} = \mathbf{C} \in {R}^{n \times D}.
        \end{equation}
    $$

    Parameters
    ----------
    n: int
        The dimension of the output space.
    D: int
        The dimension of the intermediate expansion space.
    w: torch.nn.Parameter, default = None
        The learnable parameters of the model.
        For constant reconciliation, it is assigned with a default value None.
    device: str, default = 'cpu'
        Device to perform the parameter reconciliation.

    Returns
    ----------
    torch.Tensor
        The reconciled parameter matrix of shape (n, D).
    """
    return self.c * torch.ones(n, D, device=device)