Skip to content

fabrication

Bases: Module, function

The base class of the parameter fabrication function in the tinyBIG toolkit.

It will be used as the base class template for defining the parameter reconciliation functions.

...

Notes

Formally, given the underlying data distribution mapping \(f: {R}^m \to {R}^n\) to be learned, the parameter reconciliation function \(\psi\) adjusts the available parameter vector of length \(l\) by fabricating a new parameter matrix of size \(n \times D\) to accommodate the expansion space dimension \(D\) as follows:

\[ \psi: {R}^l \to {R}^{n \times D}, \]

which is defined only on the parameters without any input data.

In most of the cases, the parameter vector length \(l\) is much smaller than the output matrix size \(n \times D\), i.e., \(l \ll n \times D\). Meanwhile, in practice, we can also define function \(\psi\) to fabricate a longer parameter vector into a smaller parameter matrix, i.e., \(l > n \times D\). To unify these different cases, the data reconciliation function can also be referred to as the "parameter fabrication function", and these function names will be used interchangeably.

Attributes:

Name Type Description
name str, default = 'base_fabrication'

Name of the parameter fabrication function.
require_parameters bool, default = True

Boolean tag of whether the function requires parameters.
enable_bias bool, default = False

Boolean tag of whether the bias is enabled or not.
device str, default = 'cpu'

Device of the parameter fabrication function.

Methods:

Name Description
__init__

It initializes the parameter fabrication function.
get_name

It gets the name of the parameter fabrication function.
calculate_l

It calculates the length of required parameters.
forward

The forward method to perform parameter fabrication.
__call__

The build-in callable method of the parameter fabrication function.
Source code in tinybig/module/base_fabrication.py 
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
class fabrication(Module, function):
    r"""
    The base class of the parameter fabrication function in the tinyBIG toolkit.

    It will be used as the base class template for defining the parameter reconciliation functions.

    ...

    Notes
    ----------
    Formally, given the underlying data distribution mapping $f: {R}^m \to {R}^n$ to be learned,
    the parameter reconciliation function $\psi$ adjusts the available parameter vector of length $l$ by fabricating
    a new parameter matrix of size $n \times D$ to accommodate the expansion space dimension $D$ as follows:

    $$ \psi: {R}^l \to {R}^{n \times D}, $$

    which is defined only on the parameters without any input data.

    In most of the cases, the parameter vector length $l$ is much smaller than the output matrix size $n \times D$,
    i.e., $l \ll n \times D$.
    Meanwhile, in practice, we can also define function $\psi$ to fabricate a longer parameter vector into a smaller
    parameter matrix, i.e., $l > n \times D$.
    To unify these different cases, the data reconciliation function can also be referred to as the
    "parameter fabrication function", and these function names will be used interchangeably.

    Attributes
    ----------
    name: str, default = 'base_fabrication'
        Name of the parameter fabrication function.
    require_parameters: bool, default = True
        Boolean tag of whether the function requires parameters.
    enable_bias: bool, default = False
        Boolean tag of whether the bias is enabled or not.
    device: str, default = 'cpu'
        Device of the parameter fabrication function.

    Methods
    ----------
    __init__
        It initializes the parameter fabrication function.

    get_name
        It gets the name of the parameter fabrication function.

    calculate_l
        It calculates the length of required parameters.

    forward
        The forward method to perform parameter fabrication.

    __call__
        The build-in callable method of the parameter fabrication function.
    """
    def __init__(
        self,
        name: str = 'base_fabrication',
        require_parameters: bool = True,
        enable_bias: bool = False,
        device: str = 'cpu',
        *args, **kwargs
    ):
        """
        The initialization method of the base parameter fabrication function.

        It initializes a base parameter fabrication function object.

        Parameters
        ----------
        name: str, default = 'base_fabrication'
            Name of the parameter fabrication function.
        require_parameters: bool, default = True
            Boolean tag of whether the function requires parameters.
        enable_bias: bool, default = False
            Boolean tag of whether the bias is enabled or not.
        device: str, default = 'cpu'
            The device of the parameter fabrication function.

        Returns
        ----------
        object
            The parameter fabrication function object.
        """
        Module.__init__(self)
        function.__init__(self, name=name, device=device)

        self.require_parameters = require_parameters
        self.enable_bias = enable_bias

    def get_name(self):
        """
        The name retrieval method of the parameter fabrication function.

        It returns the name of the parameter fabrication function.

        Returns
        -------
        str
            The name of the parameter fabrication function.
        """
        return self.name

    @abstractmethod
    def calculate_l(self, n: int, D: int):
        """
        The required parameter number calculation method.

        It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function
        based on the intermediate and output space dimensions, n and D.
        The method is declared as an abstractmethod and needs to be implemented in the inherited classes.

        Parameters
        ----------
        n: int
            The dimension of the output space.
        D: int
            The dimension of the intermediate expansion space.

        Returns
        -------
        int
            The number of required learnable parameters l.
        """
        pass

    @abstractmethod
    def forward(self, n: int, D: int, w: torch.nn.Parameter, *args, **kwargs):
        """
        The forward method of the parameter reconciliation function.

        It applies the parameter reconciliation operation to the input parameter of length l,
        and returns the reconciled parameter matrix of shape (n, D).
        The method is declared as an abstractmethod and needs to be implemented in the inherited classes.

        Parameters
        ----------
        n: int
            The dimension of the output space.
        D: int
            The dimension of the intermediate expansion space.
        w: torch.nn.Parameter
            The learnable parameters of the model of length l.

        Returns
        ----------
        torch.Tensor
            The reconciled parameter matrix of shape (n, D).
        """
        pass

__init__(name='base_fabrication', require_parameters=True, enable_bias=False, device='cpu', *args, **kwargs)

The initialization method of the base parameter fabrication function.

It initializes a base parameter fabrication function object.

Parameters:

Name Type Description Default
name str

Name of the parameter fabrication function.

 'base_fabrication'
require_parameters bool

Boolean tag of whether the function requires parameters.

 True
enable_bias bool

Boolean tag of whether the bias is enabled or not.

 False
device str

The device of the parameter fabrication function.

 'cpu'

Returns:

Type Description
object

The parameter fabrication function object.
Source code in tinybig/module/base_fabrication.py 
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
def __init__(
    self,
    name: str = 'base_fabrication',
    require_parameters: bool = True,
    enable_bias: bool = False,
    device: str = 'cpu',
    *args, **kwargs
):
    """
    The initialization method of the base parameter fabrication function.

    It initializes a base parameter fabrication function object.

    Parameters
    ----------
    name: str, default = 'base_fabrication'
        Name of the parameter fabrication function.
    require_parameters: bool, default = True
        Boolean tag of whether the function requires parameters.
    enable_bias: bool, default = False
        Boolean tag of whether the bias is enabled or not.
    device: str, default = 'cpu'
        The device of the parameter fabrication function.

    Returns
    ----------
    object
        The parameter fabrication function object.
    """
    Module.__init__(self)
    function.__init__(self, name=name, device=device)

    self.require_parameters = require_parameters
    self.enable_bias = enable_bias

calculate_l(n, D) abstractmethod

The required parameter number calculation method.

It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function based on the intermediate and output space dimensions, n and D. The method is declared as an abstractmethod and needs to be implemented in the inherited classes.

Parameters:

Name Type Description Default
n int

The dimension of the output space.

 required
D int

The dimension of the intermediate expansion space.

 required

Returns:

Type Description
int

The number of required learnable parameters l.
Source code in tinybig/module/base_fabrication.py 
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
@abstractmethod
def calculate_l(self, n: int, D: int):
    """
    The required parameter number calculation method.

    It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function
    based on the intermediate and output space dimensions, n and D.
    The method is declared as an abstractmethod and needs to be implemented in the inherited classes.

    Parameters
    ----------
    n: int
        The dimension of the output space.
    D: int
        The dimension of the intermediate expansion space.

    Returns
    -------
    int
        The number of required learnable parameters l.
    """
    pass

forward(n, D, w, *args, **kwargs) abstractmethod

The forward method of the parameter reconciliation function.

It applies the parameter reconciliation operation to the input parameter of length l, and returns the reconciled parameter matrix of shape (n, D). The method is declared as an abstractmethod and needs to be implemented in the inherited classes.

Parameters:

Name Type Description Default
n int

The dimension of the output space.

 required
D int

The dimension of the intermediate expansion space.

 required
w Parameter

The learnable parameters of the model of length l.

 required

Returns:

Type Description
Tensor

The reconciled parameter matrix of shape (n, D).
Source code in tinybig/module/base_fabrication.py 
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
@abstractmethod
def forward(self, n: int, D: int, w: torch.nn.Parameter, *args, **kwargs):
    """
    The forward method of the parameter reconciliation function.

    It applies the parameter reconciliation operation to the input parameter of length l,
    and returns the reconciled parameter matrix of shape (n, D).
    The method is declared as an abstractmethod and needs to be implemented in the inherited classes.

    Parameters
    ----------
    n: int
        The dimension of the output space.
    D: int
        The dimension of the intermediate expansion space.
    w: torch.nn.Parameter
        The learnable parameters of the model of length l.

    Returns
    ----------
    torch.Tensor
        The reconciled parameter matrix of shape (n, D).
    """
    pass

get_name()

The name retrieval method of the parameter fabrication function.

It returns the name of the parameter fabrication function.

Returns:

Type Description
str

The name of the parameter fabrication function.
Source code in tinybig/module/base_fabrication.py 
111
112
113
114
115
116
117
118
119
120
121
122
def get_name(self):
    """
    The name retrieval method of the parameter fabrication function.

    It returns the name of the parameter fabrication function.

    Returns
    -------
    str
        The name of the parameter fabrication function.
    """
    return self.name