Skip to content

grid_interdependence_layer

Bases: layer

A layer for modeling grid-based interdependence with configurable patch structures and packing strategies.

This layer supports operations over a grid input, where each head applies a grid-based interdependence function and aggregates results using optional fusion mechanisms.

Parameters:

Name Type Description Default
h int

The height of the input grid.

 required
w int

The width of the input grid.

 required
in_channel int

The number of input channels.

 required
out_channel int

The number of output channels.

 required
d int

The depth of the input grid. Default is 1 (2D grid).

 1
width int

The number of grid interdependence heads. Default is 1.

 1
name str

The name of the layer. Default is 'grid_interdependence_layer'.

 'grid_interdependence_layer'
patch_shape str

The shape of the patch ('cuboid', 'cylinder', or 'sphere'). Default is 'cuboid'.

 'cuboid'
p_h int

The height of the patch.

 None
p_h_prime int

The height of the inner patch (for hierarchical packing).

 None
p_w int

The width of the patch.

 None
p_w_prime int

The width of the inner patch (for hierarchical packing).

 None
p_d int

The depth of the patch (only for 3D grids).

 0
p_d_prime int

The depth of the inner patch (for hierarchical packing).

 None
p_r int

The radius of the patch (for spherical or cylindrical patches).

 None
cd_h int

The compression degree along the height dimension.

 None
cd_w int

The compression degree along the width dimension.

 None
cd_d int

The compression degree along the depth dimension. Default is 1.

 1
packing_strategy str

The packing strategy ('densest_packing' or other supported modes). Default is 'densest_packing'.

 'densest_packing'
with_batch_norm bool

If True, applies batch normalization. Default is True.

 True
with_relu bool

If True, applies ReLU activation. Default is True.

 True
with_residual bool

If True, includes residual connections. Default is False.

 False
enable_bias bool

If True, enables bias terms in parameter reconciliation. Default is False.

 False
with_dual_lphm bool

If True, uses dual LPHM reconciliation for parameter fabrication. Default is False.

 False
with_lorr bool

If True, uses LORR reconciliation for parameter fabrication. Default is False.

 False
r int

The rank for parameter reconciliation. Default is 3.

 3
parameters_init_method str

The initialization method for parameters. Default is 'xavier_normal'.

 'xavier_normal'
device str

The device for computations ('cpu' or 'cuda'). Default is 'cpu'.

 'cpu'
*args

Additional arguments passed to the parent class.

 ()
**kwargs

Additional arguments passed to the parent class.

 ()

Methods:

Name Description
get_output_grid_shape

Returns the shape of the output grid after packing.

Raises:

Type Description
ValueError

If invalid patch shape, dimensions, or configuration is provided.
Source code in tinybig/layer/grid_based_layers.py 
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
class grid_interdependence_layer(layer):
    """
    A layer for modeling grid-based interdependence with configurable patch structures and packing strategies.

    This layer supports operations over a grid input, where each head applies a grid-based interdependence function
    and aggregates results using optional fusion mechanisms.

    Parameters
    ----------
    h : int
        The height of the input grid.
    w : int
        The width of the input grid.
    in_channel : int
        The number of input channels.
    out_channel : int
        The number of output channels.
    d : int, optional
        The depth of the input grid. Default is 1 (2D grid).
    width : int, optional
        The number of grid interdependence heads. Default is 1.
    name : str, optional
        The name of the layer. Default is 'grid_interdependence_layer'.
    patch_shape : str, optional
        The shape of the patch ('cuboid', 'cylinder', or 'sphere'). Default is 'cuboid'.
    p_h : int, optional
        The height of the patch.
    p_h_prime : int, optional
        The height of the inner patch (for hierarchical packing).
    p_w : int, optional
        The width of the patch.
    p_w_prime : int, optional
        The width of the inner patch (for hierarchical packing).
    p_d : int, optional
        The depth of the patch (only for 3D grids).
    p_d_prime : int, optional
        The depth of the inner patch (for hierarchical packing).
    p_r : int, optional
        The radius of the patch (for spherical or cylindrical patches).
    cd_h : int, optional
        The compression degree along the height dimension.
    cd_w : int, optional
        The compression degree along the width dimension.
    cd_d : int, optional
        The compression degree along the depth dimension. Default is 1.
    packing_strategy : str, optional
        The packing strategy ('densest_packing' or other supported modes). Default is 'densest_packing'.
    with_batch_norm : bool, optional
        If True, applies batch normalization. Default is True.
    with_relu : bool, optional
        If True, applies ReLU activation. Default is True.
    with_residual : bool, optional
        If True, includes residual connections. Default is False.
    enable_bias : bool, optional
        If True, enables bias terms in parameter reconciliation. Default is False.
    with_dual_lphm : bool, optional
        If True, uses dual LPHM reconciliation for parameter fabrication. Default is False.
    with_lorr : bool, optional
        If True, uses LORR reconciliation for parameter fabrication. Default is False.
    r : int, optional
        The rank for parameter reconciliation. Default is 3.
    parameters_init_method : str, optional
        The initialization method for parameters. Default is 'xavier_normal'.
    device : str, optional
        The device for computations ('cpu' or 'cuda'). Default is 'cpu'.
    *args, **kwargs
        Additional arguments passed to the parent class.

    Methods
    -------
    get_output_grid_shape():
        Returns the shape of the output grid after packing.

    Raises
    ------
    ValueError
        If invalid patch shape, dimensions, or configuration is provided.
    """
    def __init__(
        self,
        h: int, w: int, in_channel: int, out_channel: int,
        d: int = 1,
        width: int = 1,
        name: str = 'grid_interdependence_layer',
        patch_shape: str = 'cuboid',
        p_h: int = None, p_h_prime: int = None,
        p_w: int = None, p_w_prime: int = None,
        p_d: int = 0, p_d_prime: int = None,
        p_r: int = None,
        cd_h: int = None, cd_w: int = None, cd_d: int = 1,
        packing_strategy: str = 'densest_packing',
        with_batch_norm: bool = True,
        with_relu: bool = True,
        with_residual: bool = False,
        enable_bias: bool = False,
        with_dual_lphm: bool = False,
        with_lorr: bool = False, r: int = 3,
        # other parameters
        parameters_init_method: str = 'xavier_normal',
        device: str = 'cpu', *args, **kwargs
    ):
        """
        Initialize the grid_interdependence_layer.

        This layer models interdependence across a grid using customizable patch-based structures and packing strategies.

        Parameters
        ----------
        h : int
            The height of the input grid.
        w : int
            The width of the input grid.
        in_channel : int
            The number of input channels.
        out_channel : int
            The number of output channels.
        d : int, optional
            The depth of the input grid. Default is 1 (for 2D grids).
        width : int, optional
            The number of grid interdependence heads. Default is 1.
        name : str, optional
            The name of the layer. Default is 'grid_interdependence_layer'.
        patch_shape : str, optional
            The shape of the patch ('cuboid', 'cylinder', or 'sphere'). Default is 'cuboid'.
        p_h : int, optional
            The height of the patch.
        p_h_prime : int, optional
            The height of the inner patch (used for hierarchical packing).
        p_w : int, optional
            The width of the patch.
        p_w_prime : int, optional
            The width of the inner patch (used for hierarchical packing).
        p_d : int, optional
            The depth of the patch.
        p_d_prime : int, optional
            The depth of the inner patch (used for hierarchical packing).
        p_r : int, optional
            The radius of the patch (used for spherical or cylindrical patches).
        cd_h : int, optional
            The compression degree along the height dimension.
        cd_w : int, optional
            The compression degree along the width dimension.
        cd_d : int, optional
            The compression degree along the depth dimension. Default is 1.
        packing_strategy : str, optional
            The packing strategy used ('densest_packing' or other supported strategies). Default is 'densest_packing'.
        with_batch_norm : bool, optional
            If True, applies batch normalization after the output. Default is True.
        with_relu : bool, optional
            If True, applies ReLU activation after the output. Default is True.
        with_residual : bool, optional
            If True, includes residual connections. Default is False.
        enable_bias : bool, optional
            If True, adds bias to the layer's weights. Default is False.
        with_dual_lphm : bool, optional
            If True, uses dual LPHM reconciliation for parameter fabrication. Default is False.
        with_lorr : bool, optional
            If True, uses LORR reconciliation for parameter fabrication. Default is False.
        r : int, optional
            The rank for parameter reconciliation methods. Default is 3.
        parameters_init_method : str, optional
            The initialization method for the parameters. Default is 'xavier_normal'.
        device : str, optional
            The device to use ('cpu' or 'cuda'). Default is 'cpu'.
        *args, **kwargs
            Additional arguments passed to the parent layer class.

        Raises
        ------
        ValueError
            If invalid patch shape or grid dimensions are provided.
        """
        print('* grid_interdependence_layer, width:', width)
        heads = [
            grid_interdependence_head(
                h=h, w=w, d=d,
                in_channel=in_channel, out_channel=out_channel,
                patch_shape=patch_shape,
                p_h=p_h, p_h_prime=p_h_prime,
                p_w=p_w, p_w_prime=p_w_prime,
                p_d=p_d, p_d_prime=p_d_prime,
                p_r=p_r,
                cd_h=cd_h, cd_w=cd_w, cd_d=cd_d,
                packing_strategy=packing_strategy,
                with_batch_norm=with_batch_norm,
                with_relu=with_relu,
                with_residual=with_residual,
                enable_bias=enable_bias,
                with_dual_lphm=with_dual_lphm,
                with_lorr=with_lorr, r=r,
                parameters_init_method=parameters_init_method,
                device=device, *args, **kwargs
            )
        ] * width
        assert len(heads) >= 1
        m, n = heads[0].get_m(), heads[0].get_n()
        if len(heads) > 1:
            head_fusion = mean_fusion(dims=[head.get_n() for head in heads])
        else:
            head_fusion = None
        print('--------------------------')
        super().__init__(name=name, m=m, n=n, heads=heads, head_fusion=head_fusion, device=device, *args, **kwargs)

    def get_output_grid_shape(self):
        """
        Get the shape of the output grid after packing.

        Returns
        -------
        tuple
            A tuple representing the dimensions of the output grid (height, width, depth).
        """
        assert len(self.heads) >= 1
        return self.heads[0].get_output_grid_shape()

__init__(h, w, in_channel, out_channel, d=1, width=1, name='grid_interdependence_layer', patch_shape='cuboid', p_h=None, p_h_prime=None, p_w=None, p_w_prime=None, p_d=0, p_d_prime=None, p_r=None, cd_h=None, cd_w=None, cd_d=1, packing_strategy='densest_packing', with_batch_norm=True, with_relu=True, with_residual=False, enable_bias=False, with_dual_lphm=False, with_lorr=False, r=3, parameters_init_method='xavier_normal', device='cpu', *args, **kwargs)

Initialize the grid_interdependence_layer.

This layer models interdependence across a grid using customizable patch-based structures and packing strategies.

Parameters:

Name Type Description Default
h int

The height of the input grid.

 required
w int

The width of the input grid.

 required
in_channel int

The number of input channels.

 required
out_channel int

The number of output channels.

 required
d int

The depth of the input grid. Default is 1 (for 2D grids).

 1
width int

The number of grid interdependence heads. Default is 1.

 1
name str

The name of the layer. Default is 'grid_interdependence_layer'.

 'grid_interdependence_layer'
patch_shape str

The shape of the patch ('cuboid', 'cylinder', or 'sphere'). Default is 'cuboid'.

 'cuboid'
p_h int

The height of the patch.

 None
p_h_prime int

The height of the inner patch (used for hierarchical packing).

 None
p_w int

The width of the patch.

 None
p_w_prime int

The width of the inner patch (used for hierarchical packing).

 None
p_d int

The depth of the patch.

 0
p_d_prime int

The depth of the inner patch (used for hierarchical packing).

 None
p_r int

The radius of the patch (used for spherical or cylindrical patches).

 None
cd_h int

The compression degree along the height dimension.

 None
cd_w int

The compression degree along the width dimension.

 None
cd_d int

The compression degree along the depth dimension. Default is 1.

 1
packing_strategy str

The packing strategy used ('densest_packing' or other supported strategies). Default is 'densest_packing'.

 'densest_packing'
with_batch_norm bool

If True, applies batch normalization after the output. Default is True.

 True
with_relu bool

If True, applies ReLU activation after the output. Default is True.

 True
with_residual bool

If True, includes residual connections. Default is False.

 False
enable_bias bool

If True, adds bias to the layer's weights. Default is False.

 False
with_dual_lphm bool

If True, uses dual LPHM reconciliation for parameter fabrication. Default is False.

 False
with_lorr bool

If True, uses LORR reconciliation for parameter fabrication. Default is False.

 False
r int

The rank for parameter reconciliation methods. Default is 3.

 3
parameters_init_method str

The initialization method for the parameters. Default is 'xavier_normal'.

 'xavier_normal'
device str

The device to use ('cpu' or 'cuda'). Default is 'cpu'.

 'cpu'
*args

Additional arguments passed to the parent layer class.

 ()
**kwargs

Additional arguments passed to the parent layer class.

 ()

Raises:

Type Description
ValueError

If invalid patch shape or grid dimensions are provided.
Source code in tinybig/layer/grid_based_layers.py 
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
def __init__(
    self,
    h: int, w: int, in_channel: int, out_channel: int,
    d: int = 1,
    width: int = 1,
    name: str = 'grid_interdependence_layer',
    patch_shape: str = 'cuboid',
    p_h: int = None, p_h_prime: int = None,
    p_w: int = None, p_w_prime: int = None,
    p_d: int = 0, p_d_prime: int = None,
    p_r: int = None,
    cd_h: int = None, cd_w: int = None, cd_d: int = 1,
    packing_strategy: str = 'densest_packing',
    with_batch_norm: bool = True,
    with_relu: bool = True,
    with_residual: bool = False,
    enable_bias: bool = False,
    with_dual_lphm: bool = False,
    with_lorr: bool = False, r: int = 3,
    # other parameters
    parameters_init_method: str = 'xavier_normal',
    device: str = 'cpu', *args, **kwargs
):
    """
    Initialize the grid_interdependence_layer.

    This layer models interdependence across a grid using customizable patch-based structures and packing strategies.

    Parameters
    ----------
    h : int
        The height of the input grid.
    w : int
        The width of the input grid.
    in_channel : int
        The number of input channels.
    out_channel : int
        The number of output channels.
    d : int, optional
        The depth of the input grid. Default is 1 (for 2D grids).
    width : int, optional
        The number of grid interdependence heads. Default is 1.
    name : str, optional
        The name of the layer. Default is 'grid_interdependence_layer'.
    patch_shape : str, optional
        The shape of the patch ('cuboid', 'cylinder', or 'sphere'). Default is 'cuboid'.
    p_h : int, optional
        The height of the patch.
    p_h_prime : int, optional
        The height of the inner patch (used for hierarchical packing).
    p_w : int, optional
        The width of the patch.
    p_w_prime : int, optional
        The width of the inner patch (used for hierarchical packing).
    p_d : int, optional
        The depth of the patch.
    p_d_prime : int, optional
        The depth of the inner patch (used for hierarchical packing).
    p_r : int, optional
        The radius of the patch (used for spherical or cylindrical patches).
    cd_h : int, optional
        The compression degree along the height dimension.
    cd_w : int, optional
        The compression degree along the width dimension.
    cd_d : int, optional
        The compression degree along the depth dimension. Default is 1.
    packing_strategy : str, optional
        The packing strategy used ('densest_packing' or other supported strategies). Default is 'densest_packing'.
    with_batch_norm : bool, optional
        If True, applies batch normalization after the output. Default is True.
    with_relu : bool, optional
        If True, applies ReLU activation after the output. Default is True.
    with_residual : bool, optional
        If True, includes residual connections. Default is False.
    enable_bias : bool, optional
        If True, adds bias to the layer's weights. Default is False.
    with_dual_lphm : bool, optional
        If True, uses dual LPHM reconciliation for parameter fabrication. Default is False.
    with_lorr : bool, optional
        If True, uses LORR reconciliation for parameter fabrication. Default is False.
    r : int, optional
        The rank for parameter reconciliation methods. Default is 3.
    parameters_init_method : str, optional
        The initialization method for the parameters. Default is 'xavier_normal'.
    device : str, optional
        The device to use ('cpu' or 'cuda'). Default is 'cpu'.
    *args, **kwargs
        Additional arguments passed to the parent layer class.

    Raises
    ------
    ValueError
        If invalid patch shape or grid dimensions are provided.
    """
    print('* grid_interdependence_layer, width:', width)
    heads = [
        grid_interdependence_head(
            h=h, w=w, d=d,
            in_channel=in_channel, out_channel=out_channel,
            patch_shape=patch_shape,
            p_h=p_h, p_h_prime=p_h_prime,
            p_w=p_w, p_w_prime=p_w_prime,
            p_d=p_d, p_d_prime=p_d_prime,
            p_r=p_r,
            cd_h=cd_h, cd_w=cd_w, cd_d=cd_d,
            packing_strategy=packing_strategy,
            with_batch_norm=with_batch_norm,
            with_relu=with_relu,
            with_residual=with_residual,
            enable_bias=enable_bias,
            with_dual_lphm=with_dual_lphm,
            with_lorr=with_lorr, r=r,
            parameters_init_method=parameters_init_method,
            device=device, *args, **kwargs
        )
    ] * width
    assert len(heads) >= 1
    m, n = heads[0].get_m(), heads[0].get_n()
    if len(heads) > 1:
        head_fusion = mean_fusion(dims=[head.get_n() for head in heads])
    else:
        head_fusion = None
    print('--------------------------')
    super().__init__(name=name, m=m, n=n, heads=heads, head_fusion=head_fusion, device=device, *args, **kwargs)

get_output_grid_shape()

Get the shape of the output grid after packing.

Returns:

Type Description
tuple

A tuple representing the dimensions of the output grid (height, width, depth).
Source code in tinybig/layer/grid_based_layers.py 
225
226
227
228
229
230
231
232
233
234
235
def get_output_grid_shape(self):
    """
    Get the shape of the output grid after packing.

    Returns
    -------
    tuple
        A tuple representing the dimensions of the output grid (height, width, depth).
    """
    assert len(self.heads) >= 1
    return self.heads[0].get_output_grid_shape()