Skip to content

parameterized_interdependence

Bases: interdependence

A parameterized interdependence function.

This class allows the computation of interdependence matrices parameterized by custom fabrication methods or predefined structures.

Notes

Formally, given a learnable parameter vector \(\mathbf{w} \in R^{l_{\xi}}\), the parameterized interdependence function transforms it into a matrix of desired dimensions \(m \times m'\) as follows:

\[
    \begin{equation}
    \xi(\mathbf{w}) = \text{reshape}(\mathbf{w}) = \mathbf{W} \in R^{m \times m'}.
    \end{equation}
\]

This parameterized interdependence function operates independently of any data batch, deriving the output interdependence matrix solely from the learnable parameter vector \(\mathbf{w}\), whose required length of vector \(\mathbf{w}\) is \(l_{\xi} = m \times m'\).

Attributes:

Name Type Description
parameter_fabrication Callable

A callable function or object to fabricate parameters.
b_prime int

The number of rows in the output interdependence matrix.
m_prime int

The number of columns in the output interdependence matrix.

Methods:

Name Description
calculate_l

Computes the total number of parameters needed.
calculate_b_prime

Computes the effective number of rows in the interdependence matrix.
calculate_m_prime

Computes the effective number of columns in the interdependence matrix.
calculate_A

Computes the parameterized interdependence matrix.
Source code in tinybig/interdependence/parameterized_interdependence.py 
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
class parameterized_interdependence(interdependence):
    r"""
        A parameterized interdependence function.

        This class allows the computation of interdependence matrices parameterized by custom
        fabrication methods or predefined structures.

        Notes
        ----------
        Formally, given a learnable parameter vector $\mathbf{w} \in R^{l_{\xi}}$, the parameterized interdependence function transforms it into a matrix of desired dimensions $m \times m'$ as follows:

        $$
            \begin{equation}
            \xi(\mathbf{w}) = \text{reshape}(\mathbf{w}) = \mathbf{W} \in R^{m \times m'}.
            \end{equation}
        $$

        This parameterized interdependence function operates independently of any data batch, deriving the output interdependence matrix solely from the learnable parameter vector $\mathbf{w}$,
        whose required length of vector $\mathbf{w}$ is $l_{\xi} = m \times m'$.

        Attributes
        ----------
        parameter_fabrication : Callable
            A callable function or object to fabricate parameters.
        b_prime : int
            The number of rows in the output interdependence matrix.
        m_prime : int
            The number of columns in the output interdependence matrix.

        Methods
        -------
        calculate_l()
            Computes the total number of parameters needed.
        calculate_b_prime(b=None)
            Computes the effective number of rows in the interdependence matrix.
        calculate_m_prime(m=None)
            Computes the effective number of columns in the interdependence matrix.
        calculate_A(x=None, w=None, device='cpu', *args, **kwargs)
            Computes the parameterized interdependence matrix.
    """
    def __init__(
        self,
        b: int, m: int,
        b_prime: int = None, m_prime: int = None,
        interdependence_type: str = 'attribute',
        name: str = 'parameterized_interdependence',
        require_parameters: bool = True,
        require_data: bool = False,
        device: str = 'cpu', *args, **kwargs
    ):
        """
            Initializes the parameterized interdependence function.

            Parameters
            ----------
            b : int
                Number of rows in the input tensor.
            m : int
                Number of columns in the input tensor.
            b_prime : int, optional
                Number of rows in the output interdependence matrix. Defaults to `b`.
            m_prime : int, optional
                Number of columns in the output interdependence matrix. Defaults to `m`.
            interdependence_type : str, optional
                Type of interdependence ('instance', 'attribute', etc.). Defaults to 'attribute'.
            name : str, optional
                Name of the interdependence function. Defaults to 'parameterized_interdependence'.
            require_parameters : bool, optional
                Whether parameters are required. Defaults to True.
            require_data : bool, optional
                Whether input data is required. Defaults to False.
            device : str, optional
                Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
            *args : tuple
                Additional positional arguments for the parent class.
            **kwargs : dict
                Additional keyword arguments for the parent class.
        """
        super().__init__(b=b, m=m, name=name, interdependence_type=interdependence_type, require_data=require_data, require_parameters=require_parameters, device=device, *args, **kwargs)
        self.parameter_fabrication = None
        self.b_prime = b_prime if b_prime is not None else b
        self.m_prime = m_prime if m_prime is not None else m

    def calculate_l(self):
        """
            Computes the total number of parameters required.

            Returns
            -------
            int
                The total number of parameters.

            Raises
            ------
            ValueError
                If the interdependence type is not supported.
        """
        if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
            d, d_prime = self.b, self.calculate_b_prime()
        elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
            d, d_prime = self.m, self.calculate_m_prime()
        else:
            raise ValueError(f'Interdependence type {self.interdependence_type} not supported')

        assert d is not None and d_prime is not None
        if self.parameter_fabrication is None:
            return d * d_prime
        else:
            return self.parameter_fabrication.calculate_l(n=d, D=d_prime)

    def calculate_b_prime(self, b: int = None):
        """
            Computes the effective number of rows in the interdependence matrix.

            Parameters
            ----------
            b : int, optional
                Input number of rows. Defaults to None.

            Returns
            -------
            int
                The effective number of rows in the matrix.
        """
        if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence'] and self.b_prime is not None:
            return self.b_prime
        else:
            return b if b is not None else self.b

    def calculate_m_prime(self, m: int = None):
        """
            Computes the effective number of columns in the interdependence matrix.

            Parameters
            ----------
            m : int, optional
                Input number of columns. Defaults to None.

            Returns
            -------
            int
                The effective number of columns in the matrix.
        """
        if self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence'] and self.m_prime is not None:
            return self.m_prime
        else:
            return m if m is not None else self.m

    def calculate_A(self, x: torch.Tensor = None, w: torch.nn.Parameter = None, device: str = 'cpu', *args, **kwargs):
        """
            Computes the parameterized interdependence matrix.

            Parameters
            ----------
            x : torch.Tensor, optional
                Input tensor of shape `(batch_size, num_features)`. Defaults to None.
            w : torch.nn.Parameter, optional
                Parameter tensor of shape `(num_parameters,)`. Defaults to None.
            device : str, optional
                Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
            *args : tuple
                Additional positional arguments.
            **kwargs : dict
                Additional keyword arguments.

            Returns
            -------
            torch.Tensor
                The computed interdependence matrix.

            Raises
            ------
            ValueError
                If the interdependence type is not supported.
            AssertionError
                If the parameter tensor `w` has an incorrect shape.
        """
        if not self.require_data and not self.require_parameters and self.A is not None:
            return self.A
        else:
            assert w.ndim == 2 and w.numel() == self.calculate_l()

            if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
                d, d_prime = self.b, self.calculate_b_prime()
            elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
                d, d_prime = self.m, self.calculate_m_prime()
            else:
                raise ValueError(f'Interdependence type {self.interdependence_type} not supported')

            if self.parameter_fabrication is None:
                A = w.reshape(d, d_prime).to(device=device)
            else:
                A = self.parameter_fabrication(w=w, n=d, D=d_prime, device=device)
            A = self.post_process(x=A, device=device)

            if self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
                assert A.shape == (self.m, self.calculate_m_prime())
            elif self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
                assert A.shape == (self.b, self.calculate_b_prime())

            if not self.require_data and not self.require_parameters and self.A is None:
                self.A = A
            return A

__init__(b, m, b_prime=None, m_prime=None, interdependence_type='attribute', name='parameterized_interdependence', require_parameters=True, require_data=False, device='cpu', *args, **kwargs)

Initializes the parameterized interdependence function.

Parameters:

Name Type Description Default
b int

Number of rows in the input tensor.

 required
m int

Number of columns in the input tensor.

 required
b_prime int

Number of rows in the output interdependence matrix. Defaults to b.

 None
m_prime int

Number of columns in the output interdependence matrix. Defaults to m.

 None
interdependence_type str

Type of interdependence ('instance', 'attribute', etc.). Defaults to 'attribute'.

 'attribute'
name str

Name of the interdependence function. Defaults to 'parameterized_interdependence'.

 'parameterized_interdependence'
require_parameters bool

Whether parameters are required. Defaults to True.

 True
require_data bool

Whether input data is required. Defaults to False.

 False
device str

Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.

 'cpu'
*args tuple

Additional positional arguments for the parent class.

 ()
**kwargs dict

Additional keyword arguments for the parent class.

 {}
Source code in tinybig/interdependence/parameterized_interdependence.py 
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
def __init__(
    self,
    b: int, m: int,
    b_prime: int = None, m_prime: int = None,
    interdependence_type: str = 'attribute',
    name: str = 'parameterized_interdependence',
    require_parameters: bool = True,
    require_data: bool = False,
    device: str = 'cpu', *args, **kwargs
):
    """
        Initializes the parameterized interdependence function.

        Parameters
        ----------
        b : int
            Number of rows in the input tensor.
        m : int
            Number of columns in the input tensor.
        b_prime : int, optional
            Number of rows in the output interdependence matrix. Defaults to `b`.
        m_prime : int, optional
            Number of columns in the output interdependence matrix. Defaults to `m`.
        interdependence_type : str, optional
            Type of interdependence ('instance', 'attribute', etc.). Defaults to 'attribute'.
        name : str, optional
            Name of the interdependence function. Defaults to 'parameterized_interdependence'.
        require_parameters : bool, optional
            Whether parameters are required. Defaults to True.
        require_data : bool, optional
            Whether input data is required. Defaults to False.
        device : str, optional
            Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
        *args : tuple
            Additional positional arguments for the parent class.
        **kwargs : dict
            Additional keyword arguments for the parent class.
    """
    super().__init__(b=b, m=m, name=name, interdependence_type=interdependence_type, require_data=require_data, require_parameters=require_parameters, device=device, *args, **kwargs)
    self.parameter_fabrication = None
    self.b_prime = b_prime if b_prime is not None else b
    self.m_prime = m_prime if m_prime is not None else m

calculate_A(x=None, w=None, device='cpu', *args, **kwargs)

Computes the parameterized interdependence matrix.

Parameters:

Name Type Description Default
x Tensor

Input tensor of shape (batch_size, num_features). Defaults to None.

 None
w Parameter

Parameter tensor of shape (num_parameters,). Defaults to None.

 None
device str

Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.

 'cpu'
*args tuple

Additional positional arguments.

 ()
**kwargs dict

Additional keyword arguments.

 {}

Returns:

Type Description
Tensor

The computed interdependence matrix.

Raises:

Type Description
ValueError

If the interdependence type is not supported.
AssertionError

If the parameter tensor w has an incorrect shape.
Source code in tinybig/interdependence/parameterized_interdependence.py 
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
def calculate_A(self, x: torch.Tensor = None, w: torch.nn.Parameter = None, device: str = 'cpu', *args, **kwargs):
    """
        Computes the parameterized interdependence matrix.

        Parameters
        ----------
        x : torch.Tensor, optional
            Input tensor of shape `(batch_size, num_features)`. Defaults to None.
        w : torch.nn.Parameter, optional
            Parameter tensor of shape `(num_parameters,)`. Defaults to None.
        device : str, optional
            Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
        *args : tuple
            Additional positional arguments.
        **kwargs : dict
            Additional keyword arguments.

        Returns
        -------
        torch.Tensor
            The computed interdependence matrix.

        Raises
        ------
        ValueError
            If the interdependence type is not supported.
        AssertionError
            If the parameter tensor `w` has an incorrect shape.
    """
    if not self.require_data and not self.require_parameters and self.A is not None:
        return self.A
    else:
        assert w.ndim == 2 and w.numel() == self.calculate_l()

        if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
            d, d_prime = self.b, self.calculate_b_prime()
        elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
            d, d_prime = self.m, self.calculate_m_prime()
        else:
            raise ValueError(f'Interdependence type {self.interdependence_type} not supported')

        if self.parameter_fabrication is None:
            A = w.reshape(d, d_prime).to(device=device)
        else:
            A = self.parameter_fabrication(w=w, n=d, D=d_prime, device=device)
        A = self.post_process(x=A, device=device)

        if self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
            assert A.shape == (self.m, self.calculate_m_prime())
        elif self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
            assert A.shape == (self.b, self.calculate_b_prime())

        if not self.require_data and not self.require_parameters and self.A is None:
            self.A = A
        return A

calculate_b_prime(b=None)

Computes the effective number of rows in the interdependence matrix.

Parameters:

Name Type Description Default
b int

Input number of rows. Defaults to None.

 None

Returns:

Type Description
int

The effective number of rows in the matrix.
Source code in tinybig/interdependence/parameterized_interdependence.py 
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
def calculate_b_prime(self, b: int = None):
    """
        Computes the effective number of rows in the interdependence matrix.

        Parameters
        ----------
        b : int, optional
            Input number of rows. Defaults to None.

        Returns
        -------
        int
            The effective number of rows in the matrix.
    """
    if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence'] and self.b_prime is not None:
        return self.b_prime
    else:
        return b if b is not None else self.b

calculate_l()

Computes the total number of parameters required.

Returns:

Type Description
int

The total number of parameters.

Raises:

Type Description
ValueError

If the interdependence type is not supported.
Source code in tinybig/interdependence/parameterized_interdependence.py 
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
def calculate_l(self):
    """
        Computes the total number of parameters required.

        Returns
        -------
        int
            The total number of parameters.

        Raises
        ------
        ValueError
            If the interdependence type is not supported.
    """
    if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
        d, d_prime = self.b, self.calculate_b_prime()
    elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
        d, d_prime = self.m, self.calculate_m_prime()
    else:
        raise ValueError(f'Interdependence type {self.interdependence_type} not supported')

    assert d is not None and d_prime is not None
    if self.parameter_fabrication is None:
        return d * d_prime
    else:
        return self.parameter_fabrication.calculate_l(n=d, D=d_prime)

calculate_m_prime(m=None)

Computes the effective number of columns in the interdependence matrix.

Parameters:

Name Type Description Default
m int

Input number of columns. Defaults to None.

 None

Returns:

Type Description
int

The effective number of columns in the matrix.
Source code in tinybig/interdependence/parameterized_interdependence.py 
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
def calculate_m_prime(self, m: int = None):
    """
        Computes the effective number of columns in the interdependence matrix.

        Parameters
        ----------
        m : int, optional
            Input number of columns. Defaults to None.

        Returns
        -------
        int
            The effective number of columns in the matrix.
    """
    if self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence'] and self.m_prime is not None:
        return self.m_prime
    else:
        return m if m is not None else self.m