Skip to content

perceptron_layer

Bases: layer

A layer consisting of perceptron heads.

This layer uses perceptron heads with optional data expansion, parameter reconciliation, and output processing.

Parameters:

Name Type Description Default
m int

The input dimension of the perceptron heads.

 required
n int

The output dimension of the perceptron heads.

 required
name str

The name of the layer.

 'perceptron_layer'
channel_num int

The number of channels in the layer.

 1
width int

The number of perceptron heads in the layer.

 1
with_bspline bool

Whether to use B-spline expansion.

 False
with_taylor bool

Whether to use Taylor expansion.

 False
d int

The degree of expansion when using Taylor or B-spline expansion.

 2
with_hybrid_expansion bool

Whether to use hybrid expansion.

 False
with_dual_lphm bool

Whether to use dual LPHM reconciliation.

 False
with_lorr bool

Whether to use LORR reconciliation.

 False
r int

The rank for parameter reconciliation.

 3
enable_bias bool

Whether to enable bias in parameter reconciliation.

 True
with_residual bool

Whether to include a residual connection.

 False
with_batch_norm bool

Whether to apply batch normalization.

 False
with_relu bool

Whether to apply ReLU activation.

 True
with_dropout bool

Whether to apply dropout.

 True
p float

Dropout probability.

 0.5
with_softmax bool

Whether to apply softmax activation.

 True
parameters_init_method str

The method for parameter initialization.

 'xavier_normal'
device str

The device to run the layer on ('cpu' or 'cuda').

 'cpu'

Returns:

Type Description
perceptron_layer

An initialized perceptron layer with the specified configuration.
Source code in tinybig/layer/basic_layers.py 
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
class perceptron_layer(layer):
    """
    A layer consisting of perceptron heads.

    This layer uses perceptron heads with optional data expansion, parameter reconciliation, and output processing.

    Parameters
    ----------
    m : int
        The input dimension of the perceptron heads.
    n : int
        The output dimension of the perceptron heads.
    name : str, default='perceptron_layer'
        The name of the layer.
    channel_num : int, default=1
        The number of channels in the layer.
    width : int, default=1
        The number of perceptron heads in the layer.
    with_bspline : bool, default=False
        Whether to use B-spline expansion.
    with_taylor : bool, default=False
        Whether to use Taylor expansion.
    d : int, default=2
        The degree of expansion when using Taylor or B-spline expansion.
    with_hybrid_expansion : bool, default=False
        Whether to use hybrid expansion.
    with_dual_lphm : bool, default=False
        Whether to use dual LPHM reconciliation.
    with_lorr : bool, default=False
        Whether to use LORR reconciliation.
    r : int, default=3
        The rank for parameter reconciliation.
    enable_bias : bool, default=True
        Whether to enable bias in parameter reconciliation.
    with_residual : bool, default=False
        Whether to include a residual connection.
    with_batch_norm : bool, default=False
        Whether to apply batch normalization.
    with_relu : bool, default=True
        Whether to apply ReLU activation.
    with_dropout : bool, default=True
        Whether to apply dropout.
    p : float, default=0.5
        Dropout probability.
    with_softmax : bool, default=True
        Whether to apply softmax activation.
    parameters_init_method : str, default='xavier_normal'
        The method for parameter initialization.
    device : str, default='cpu'
        The device to run the layer on ('cpu' or 'cuda').

    Returns
    -------
    perceptron_layer
        An initialized perceptron layer with the specified configuration.
    """

    def __init__(
        self,
        m: int, n: int,
        name: str = 'perceptron_layer',
        channel_num: int = 1,
        width: int = 1,
        # data expansion function
        with_bspline: bool = False,
        with_taylor: bool = False, d: int = 2,
        with_hybrid_expansion: bool = False,
        # parameter reconciliation function parameters
        with_dual_lphm: bool = False,
        with_lorr: bool = False, r: int = 3,
        enable_bias: bool = True,
        # remainder function parameters
        with_residual: bool = False,
        # output processing function parameters
        with_batch_norm: bool = False,
        with_relu: bool = True,
        with_dropout: bool = True, p: float = 0.5,
        with_softmax: bool = True,
        # other parameters
        parameters_init_method: str = 'xavier_normal',
        device: str = 'cpu', *args, **kwargs
    ):
        """
        Initialize a perceptron layer.

        Parameters
        ----------
        m : int
            The input dimension of the perceptron heads.
        n : int
            The output dimension of the perceptron heads.
        name : str, default='perceptron_layer'
            The name of the layer.
        channel_num : int, default=1
            The number of channels in the layer.
        width : int, default=1
            The number of perceptron heads in the layer.
        with_bspline : bool, default=False
            Whether to use B-spline expansion.
        with_taylor : bool, default=False
            Whether to use Taylor expansion.
        d : int, default=2
            The degree of expansion when using Taylor or B-spline expansion.
        with_hybrid_expansion : bool, default=False
            Whether to use hybrid expansion.
        with_dual_lphm : bool, default=False
            Whether to use dual LPHM reconciliation.
        with_lorr : bool, default=False
            Whether to use LORR reconciliation.
        r : int, default=3
            The rank for parameter reconciliation.
        enable_bias : bool, default=True
            Whether to enable bias in parameter reconciliation.
        with_residual : bool, default=False
            Whether to include a residual connection.
        with_batch_norm : bool, default=False
            Whether to apply batch normalization.
        with_relu : bool, default=True
            Whether to apply ReLU activation.
        with_dropout : bool, default=True
            Whether to apply dropout.
        p : float, default=0.5
            Dropout probability.
        with_softmax : bool, default=True
            Whether to apply softmax activation.
        parameters_init_method : str, default='xavier_normal'
            The method for parameter initialization.
        device : str, default='cpu'
            The device to run the layer on ('cpu' or 'cuda').

        Returns
        -------
        None
        """
        print('* perceptron_layer, width:', width)
        heads = [
            perceptron_head(
                m=m, n=n,
                channel_num=channel_num,
                # --------------------
                with_bspline=with_bspline,
                with_taylor=with_taylor, d=d,
                with_hybrid_expansion=with_hybrid_expansion,
                # --------------------
                with_dual_lphm=with_dual_lphm,
                with_lorr=with_lorr, r=r,
                enable_bias=enable_bias,
                # --------------------
                with_residual=with_residual,
                # --------------------
                with_batch_norm=with_batch_norm,
                with_relu=with_relu,
                with_dropout=with_dropout, p=p,
                with_softmax=with_softmax,
                # --------------------
                parameters_init_method=parameters_init_method,
                device=device, *args, **kwargs
            )
        ] * width
        print('--------------------------')
        super().__init__(name=name, m=m, n=n, heads=heads, device=device, *args, **kwargs)

__init__(m, n, name='perceptron_layer', channel_num=1, width=1, with_bspline=False, with_taylor=False, d=2, with_hybrid_expansion=False, with_dual_lphm=False, with_lorr=False, r=3, enable_bias=True, with_residual=False, with_batch_norm=False, with_relu=True, with_dropout=True, p=0.5, with_softmax=True, parameters_init_method='xavier_normal', device='cpu', *args, **kwargs)

Initialize a perceptron layer.

Parameters:

Name Type Description Default
m int

The input dimension of the perceptron heads.

 required
n int

The output dimension of the perceptron heads.

 required
name str

The name of the layer.

 'perceptron_layer'
channel_num int

The number of channels in the layer.

 1
width int

The number of perceptron heads in the layer.

 1
with_bspline bool

Whether to use B-spline expansion.

 False
with_taylor bool

Whether to use Taylor expansion.

 False
d int

The degree of expansion when using Taylor or B-spline expansion.

 2
with_hybrid_expansion bool

Whether to use hybrid expansion.

 False
with_dual_lphm bool

Whether to use dual LPHM reconciliation.

 False
with_lorr bool

Whether to use LORR reconciliation.

 False
r int

The rank for parameter reconciliation.

 3
enable_bias bool

Whether to enable bias in parameter reconciliation.

 True
with_residual bool

Whether to include a residual connection.

 False
with_batch_norm bool

Whether to apply batch normalization.

 False
with_relu bool

Whether to apply ReLU activation.

 True
with_dropout bool

Whether to apply dropout.

 True
p float

Dropout probability.

 0.5
with_softmax bool

Whether to apply softmax activation.

 True
parameters_init_method str

The method for parameter initialization.

 'xavier_normal'
device str

The device to run the layer on ('cpu' or 'cuda').

 'cpu'

Returns:

Type Description
None
Source code in tinybig/layer/basic_layers.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
def __init__(
    self,
    m: int, n: int,
    name: str = 'perceptron_layer',
    channel_num: int = 1,
    width: int = 1,
    # data expansion function
    with_bspline: bool = False,
    with_taylor: bool = False, d: int = 2,
    with_hybrid_expansion: bool = False,
    # parameter reconciliation function parameters
    with_dual_lphm: bool = False,
    with_lorr: bool = False, r: int = 3,
    enable_bias: bool = True,
    # remainder function parameters
    with_residual: bool = False,
    # output processing function parameters
    with_batch_norm: bool = False,
    with_relu: bool = True,
    with_dropout: bool = True, p: float = 0.5,
    with_softmax: bool = True,
    # other parameters
    parameters_init_method: str = 'xavier_normal',
    device: str = 'cpu', *args, **kwargs
):
    """
    Initialize a perceptron layer.

    Parameters
    ----------
    m : int
        The input dimension of the perceptron heads.
    n : int
        The output dimension of the perceptron heads.
    name : str, default='perceptron_layer'
        The name of the layer.
    channel_num : int, default=1
        The number of channels in the layer.
    width : int, default=1
        The number of perceptron heads in the layer.
    with_bspline : bool, default=False
        Whether to use B-spline expansion.
    with_taylor : bool, default=False
        Whether to use Taylor expansion.
    d : int, default=2
        The degree of expansion when using Taylor or B-spline expansion.
    with_hybrid_expansion : bool, default=False
        Whether to use hybrid expansion.
    with_dual_lphm : bool, default=False
        Whether to use dual LPHM reconciliation.
    with_lorr : bool, default=False
        Whether to use LORR reconciliation.
    r : int, default=3
        The rank for parameter reconciliation.
    enable_bias : bool, default=True
        Whether to enable bias in parameter reconciliation.
    with_residual : bool, default=False
        Whether to include a residual connection.
    with_batch_norm : bool, default=False
        Whether to apply batch normalization.
    with_relu : bool, default=True
        Whether to apply ReLU activation.
    with_dropout : bool, default=True
        Whether to apply dropout.
    p : float, default=0.5
        Dropout probability.
    with_softmax : bool, default=True
        Whether to apply softmax activation.
    parameters_init_method : str, default='xavier_normal'
        The method for parameter initialization.
    device : str, default='cpu'
        The device to run the layer on ('cpu' or 'cuda').

    Returns
    -------
    None
    """
    print('* perceptron_layer, width:', width)
    heads = [
        perceptron_head(
            m=m, n=n,
            channel_num=channel_num,
            # --------------------
            with_bspline=with_bspline,
            with_taylor=with_taylor, d=d,
            with_hybrid_expansion=with_hybrid_expansion,
            # --------------------
            with_dual_lphm=with_dual_lphm,
            with_lorr=with_lorr, r=r,
            enable_bias=enable_bias,
            # --------------------
            with_residual=with_residual,
            # --------------------
            with_batch_norm=with_batch_norm,
            with_relu=with_relu,
            with_dropout=with_dropout, p=p,
            with_softmax=with_softmax,
            # --------------------
            parameters_init_method=parameters_init_method,
            device=device, *args, **kwargs
        )
    ] * width
    print('--------------------------')
    super().__init__(name=name, m=m, n=n, heads=heads, device=device, *args, **kwargs)