Skip to content

parameterized_bilinear_interdependence

Bases: interdependence

A parameterized bilinear interdependence function.

This class computes interdependence matrices using a bilinear transformation parameterized by custom fabrication methods or predefined structures.

Notes

Formally, given a data batch \(\mathbf{X} \in R^{b \times m}\), we can represent the parameterized bilinear form-based interdependence function as follows:

\[
    \begin{equation}\label{equ:bilinear_interdependence_function}
    \xi(\mathbf{X} | \mathbf{w}) = \mathbf{X}^\top \mathbf{W} \mathbf{X} = \mathbf{A} \in R^{m \times m},
    \end{equation}
\]

where \(\mathbf{W} = \text{reshape}(\mathbf{w}) \in R^{b \times b}\) denotes the parameter matrix reshaped from the learnable parameter vector \(\mathbf{w} \in R^{l_{\xi}}\).

The required length of parameter vector of this interdependence function is \(l_{\xi} = b^2\).

Attributes:

Name Type Description
parameter_fabrication Callable

A callable function or object to fabricate parameters.

Methods:

Name Description
calculate_b_prime

Computes the effective number of rows in the interdependence matrix.
calculate_m_prime

Computes the effective number of columns in the interdependence matrix.
calculate_l

Computes the total number of parameters needed.
calculate_A

Computes the parameterized bilinear interdependence matrix.
Source code in tinybig/interdependence/parameterized_bilinear_interdependence.py 
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
class parameterized_bilinear_interdependence(interdependence):
    r"""
        A parameterized bilinear interdependence function.

        This class computes interdependence matrices using a bilinear transformation
        parameterized by custom fabrication methods or predefined structures.

        Notes
        ----------
        Formally, given a data batch $\mathbf{X} \in R^{b \times m}$, we can represent the parameterized bilinear form-based interdependence function as follows:

        $$
            \begin{equation}\label{equ:bilinear_interdependence_function}
            \xi(\mathbf{X} | \mathbf{w}) = \mathbf{X}^\top \mathbf{W} \mathbf{X} = \mathbf{A} \in R^{m \times m},
            \end{equation}
        $$

        where $\mathbf{W} = \text{reshape}(\mathbf{w}) \in R^{b \times b}$ denotes the parameter matrix reshaped from the learnable parameter vector $\mathbf{w} \in R^{l_{\xi}}$.

        The required length of parameter vector of this interdependence function is $l_{\xi} = b^2$.


        Attributes
        ----------
        parameter_fabrication : Callable
            A callable function or object to fabricate parameters.

        Methods
        -------
        calculate_b_prime(b=None)
            Computes the effective number of rows in the interdependence matrix.
        calculate_m_prime(m=None)
            Computes the effective number of columns in the interdependence matrix.
        calculate_l()
            Computes the total number of parameters needed.
        calculate_A(x=None, w=None, device='cpu', *args, **kwargs)
            Computes the parameterized bilinear interdependence matrix.
    """

    def __init__(
        self,
        b: int, m: int,
        interdependence_type: str = 'instance',
        name: str = 'parameterized_bilinear_interdependence',
        require_parameters: bool = True,
        require_data: bool = True,
        device: str = 'cpu', *args, **kwargs
    ):
        """
            Initializes the parameterized bilinear interdependence function.

            Parameters
            ----------
            b : int
                Number of rows in the input tensor.
            m : int
                Number of columns in the input tensor.
            interdependence_type : str, optional
                Type of interdependence ('instance', 'attribute', etc.). Defaults to 'instance'.
            name : str, optional
                Name of the interdependence function. Defaults to 'parameterized_bilinear_interdependence'.
            require_parameters : bool, optional
                Whether parameters are required. Defaults to True.
            require_data : bool, optional
                Whether input data is required. Defaults to True.
            device : str, optional
                Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
            *args : tuple
                Additional positional arguments for the parent class.
            **kwargs : dict
                Additional keyword arguments for the parent class.
        """
        super().__init__(b=b, m=m, name=name, interdependence_type=interdependence_type, require_data=require_data, require_parameters=require_parameters, device=device, *args, **kwargs)
        self.parameter_fabrication = None

    def calculate_b_prime(self, b: int = None):
        """
            Computes the effective number of rows in the interdependence matrix.

            Parameters
            ----------
            b : int, optional
                Input number of rows. Defaults to None.

            Returns
            -------
            int
                The effective number of rows in the matrix.
        """
        b = b if b is not None else self.b
        return b

    def calculate_m_prime(self, m: int = None):
        """
            Computes the effective number of columns in the interdependence matrix.

            Parameters
            ----------
            m : int, optional
                Input number of columns. Defaults to None.

            Returns
            -------
            int
                The effective number of columns in the matrix.
        """
        m = m if m is not None else self.m
        return m

    def calculate_l(self):
        """
            Computes the total number of parameters required.

            Returns
            -------
            int
                The total number of parameters.

            Raises
            ------
            ValueError
                If the interdependence type is not supported.
        """
        if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
            if self.parameter_fabrication is None:
                return self.m ** 2
            else:
                return self.parameter_fabrication.calculate_l(n=self.m, D=self.m)
        elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
            if self.parameter_fabrication is None:
                return self.b ** 2
            else:
                return self.parameter_fabrication.calculate_l(n=self.b, D=self.b)
        else:
            raise ValueError(f'Interdependence type {self.interdependence_type} not supported')

    def calculate_A(self, x: torch.Tensor = None, w: torch.nn.Parameter = None, device: str = 'cpu', *args, **kwargs):
        """
            Computes the parameterized bilinear interdependence matrix.

            Parameters
            ----------
            x : torch.Tensor, optional
                Input tensor of shape `(batch_size, num_features)`. Required for computation.
            w : torch.nn.Parameter, optional
                Parameter tensor of shape `(num_parameters,)`. Required for computation.
            device : str, optional
                Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
            *args : tuple
                Additional positional arguments.
            **kwargs : dict
                Additional keyword arguments.

            Returns
            -------
            torch.Tensor
                The computed interdependence matrix.

            Raises
            ------
            ValueError
                If the interdependence type is not supported.
            AssertionError
                If input data or parameter tensor `w` has an incorrect shape.
        """
        if not self.require_data and not self.require_parameters and self.A is not None:
            return self.A
        else:
            assert x is not None and x.ndim == 2
            assert w is not None and w.ndim == 2 and w.numel() == self.calculate_l()

            x = self.pre_process(x=x, device=device)

            if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
                # for instance interdependence, the parameter for calculating x.t*W*x will have dimension m*m'
                d, d_prime = self.m, self.calculate_m_prime()
            elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
                # for attribute interdependence, the parameter for calculating x.t*W*x will have dimension b*b'
                d, d_prime = self.b, self.calculate_b_prime()
            else:
                raise ValueError(f'Interdependence type {self.interdependence_type} not supported')

            if self.parameter_fabrication is None:
                W = w.reshape(d, d_prime).to(device=device)
            else:
                W = self.parameter_fabrication(w=w, n=d, D=d_prime, device=device)

            A = torch.matmul(x.t(), torch.matmul(W, x))
            A = self.post_process(x=A, device=device)

            # if self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
            #     assert A.shape == (self.m, self.calculate_m_prime())
            # elif self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
            #     assert A.shape == (self.b, self.calculate_b_prime())

            if not self.require_data and not self.require_parameters and self.A is None:
                self.A = A
            return A

__init__(b, m, interdependence_type='instance', name='parameterized_bilinear_interdependence', require_parameters=True, require_data=True, device='cpu', *args, **kwargs)

Initializes the parameterized bilinear interdependence function.

Parameters:

Name Type Description Default
b int

Number of rows in the input tensor.

 required
m int

Number of columns in the input tensor.

 required
interdependence_type str

Type of interdependence ('instance', 'attribute', etc.). Defaults to 'instance'.

 'instance'
name str

Name of the interdependence function. Defaults to 'parameterized_bilinear_interdependence'.

 'parameterized_bilinear_interdependence'
require_parameters bool

Whether parameters are required. Defaults to True.

 True
require_data bool

Whether input data is required. Defaults to True.

 True
device str

Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.

 'cpu'
*args tuple

Additional positional arguments for the parent class.

 ()
**kwargs dict

Additional keyword arguments for the parent class.

 {}
Source code in tinybig/interdependence/parameterized_bilinear_interdependence.py 
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
def __init__(
    self,
    b: int, m: int,
    interdependence_type: str = 'instance',
    name: str = 'parameterized_bilinear_interdependence',
    require_parameters: bool = True,
    require_data: bool = True,
    device: str = 'cpu', *args, **kwargs
):
    """
        Initializes the parameterized bilinear interdependence function.

        Parameters
        ----------
        b : int
            Number of rows in the input tensor.
        m : int
            Number of columns in the input tensor.
        interdependence_type : str, optional
            Type of interdependence ('instance', 'attribute', etc.). Defaults to 'instance'.
        name : str, optional
            Name of the interdependence function. Defaults to 'parameterized_bilinear_interdependence'.
        require_parameters : bool, optional
            Whether parameters are required. Defaults to True.
        require_data : bool, optional
            Whether input data is required. Defaults to True.
        device : str, optional
            Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
        *args : tuple
            Additional positional arguments for the parent class.
        **kwargs : dict
            Additional keyword arguments for the parent class.
    """
    super().__init__(b=b, m=m, name=name, interdependence_type=interdependence_type, require_data=require_data, require_parameters=require_parameters, device=device, *args, **kwargs)
    self.parameter_fabrication = None

calculate_A(x=None, w=None, device='cpu', *args, **kwargs)

Computes the parameterized bilinear interdependence matrix.

Parameters:

Name Type Description Default
x Tensor

Input tensor of shape (batch_size, num_features). Required for computation.

 None
w Parameter

Parameter tensor of shape (num_parameters,). Required for computation.

 None
device str

Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.

 'cpu'
*args tuple

Additional positional arguments.

 ()
**kwargs dict

Additional keyword arguments.

 {}

Returns:

Type Description
Tensor

The computed interdependence matrix.

Raises:

Type Description
ValueError

If the interdependence type is not supported.
AssertionError

If input data or parameter tensor w has an incorrect shape.
Source code in tinybig/interdependence/parameterized_bilinear_interdependence.py 
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
def calculate_A(self, x: torch.Tensor = None, w: torch.nn.Parameter = None, device: str = 'cpu', *args, **kwargs):
    """
        Computes the parameterized bilinear interdependence matrix.

        Parameters
        ----------
        x : torch.Tensor, optional
            Input tensor of shape `(batch_size, num_features)`. Required for computation.
        w : torch.nn.Parameter, optional
            Parameter tensor of shape `(num_parameters,)`. Required for computation.
        device : str, optional
            Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
        *args : tuple
            Additional positional arguments.
        **kwargs : dict
            Additional keyword arguments.

        Returns
        -------
        torch.Tensor
            The computed interdependence matrix.

        Raises
        ------
        ValueError
            If the interdependence type is not supported.
        AssertionError
            If input data or parameter tensor `w` has an incorrect shape.
    """
    if not self.require_data and not self.require_parameters and self.A is not None:
        return self.A
    else:
        assert x is not None and x.ndim == 2
        assert w is not None and w.ndim == 2 and w.numel() == self.calculate_l()

        x = self.pre_process(x=x, device=device)

        if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
            # for instance interdependence, the parameter for calculating x.t*W*x will have dimension m*m'
            d, d_prime = self.m, self.calculate_m_prime()
        elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
            # for attribute interdependence, the parameter for calculating x.t*W*x will have dimension b*b'
            d, d_prime = self.b, self.calculate_b_prime()
        else:
            raise ValueError(f'Interdependence type {self.interdependence_type} not supported')

        if self.parameter_fabrication is None:
            W = w.reshape(d, d_prime).to(device=device)
        else:
            W = self.parameter_fabrication(w=w, n=d, D=d_prime, device=device)

        A = torch.matmul(x.t(), torch.matmul(W, x))
        A = self.post_process(x=A, device=device)

        # if self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
        #     assert A.shape == (self.m, self.calculate_m_prime())
        # elif self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
        #     assert A.shape == (self.b, self.calculate_b_prime())

        if not self.require_data and not self.require_parameters and self.A is None:
            self.A = A
        return A

calculate_b_prime(b=None)

Computes the effective number of rows in the interdependence matrix.

Parameters:

Name Type Description Default
b int

Input number of rows. Defaults to None.

 None

Returns:

Type Description
int

The effective number of rows in the matrix.
Source code in tinybig/interdependence/parameterized_bilinear_interdependence.py 
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
def calculate_b_prime(self, b: int = None):
    """
        Computes the effective number of rows in the interdependence matrix.

        Parameters
        ----------
        b : int, optional
            Input number of rows. Defaults to None.

        Returns
        -------
        int
            The effective number of rows in the matrix.
    """
    b = b if b is not None else self.b
    return b

calculate_l()

Computes the total number of parameters required.

Returns:

Type Description
int

The total number of parameters.

Raises:

Type Description
ValueError

If the interdependence type is not supported.
Source code in tinybig/interdependence/parameterized_bilinear_interdependence.py 
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
def calculate_l(self):
    """
        Computes the total number of parameters required.

        Returns
        -------
        int
            The total number of parameters.

        Raises
        ------
        ValueError
            If the interdependence type is not supported.
    """
    if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
        if self.parameter_fabrication is None:
            return self.m ** 2
        else:
            return self.parameter_fabrication.calculate_l(n=self.m, D=self.m)
    elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
        if self.parameter_fabrication is None:
            return self.b ** 2
        else:
            return self.parameter_fabrication.calculate_l(n=self.b, D=self.b)
    else:
        raise ValueError(f'Interdependence type {self.interdependence_type} not supported')

calculate_m_prime(m=None)

Computes the effective number of columns in the interdependence matrix.

Parameters:

Name Type Description Default
m int

Input number of columns. Defaults to None.

 None

Returns:

Type Description
int

The effective number of columns in the matrix.
Source code in tinybig/interdependence/parameterized_bilinear_interdependence.py 
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
def calculate_m_prime(self, m: int = None):
    """
        Computes the effective number of columns in the interdependence matrix.

        Parameters
        ----------
        m : int, optional
            Input number of columns. Defaults to None.

        Returns
        -------
        int
            The effective number of columns in the matrix.
    """
    m = m if m is not None else self.m
    return m