Skip to content

mlp

Bases: rpn

A Multi-Layer Perceptron (MLP) implemented using the RPN framework.

Parameters:

Name Type Description Default
dims list[int] | tuple[int]

A list or tuple of integers representing the dimensions of each layer in the MLP. Must contain at least two dimensions.

 required
name str

The name of the MLP model. Default is 'rpn_mlp'.

 'rpn_mlp'
enable_bias bool

Whether to enable bias in the layers. Default is False.

 False
with_taylor bool

Whether to use Taylor expansion for data transformation. Default is False.

 False
d int

The degree of the Taylor expansion if with_taylor is True. Default is 2.

 2
with_lorr bool

Whether to use LoRR (Low-Rank Representation) for parameter reconciliation. Default is False.

 False
r int

The rank for the LoRR parameter reconciliation. Default is 3.

 3
with_residual bool

Whether to enable residual connections in the layers. Default is False.

 False
channel_num int

The number of channels in each layer. Default is 1.

 1
width int

The number of parallel heads in each layer. Default is 1.

 1
device str

The device to use for computation ('cpu' or 'cuda'). Default is 'cpu'.

 'cpu'
*args optional

Additional positional arguments for the rpn superclass.

 ()
**kwargs optional

Additional keyword arguments for the rpn superclass.

 {}

Raises:

Type Description
ValueError

If dims contains fewer than two dimensions.

Methods:

Name Description
__init__

Initializes the MLP model and builds its layers.
Source code in tinybig/model/rpn_mlp.py 
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
class mlp(rpn):
    """
    A Multi-Layer Perceptron (MLP) implemented using the RPN framework.

    Parameters
    ----------
    dims : list[int] | tuple[int]
        A list or tuple of integers representing the dimensions of each layer in the MLP.
        Must contain at least two dimensions.
    name : str, optional
        The name of the MLP model. Default is 'rpn_mlp'.
    enable_bias : bool, optional
        Whether to enable bias in the layers. Default is False.
    with_taylor : bool, optional
        Whether to use Taylor expansion for data transformation. Default is False.
    d : int, optional
        The degree of the Taylor expansion if `with_taylor` is True. Default is 2.
    with_lorr : bool, optional
        Whether to use LoRR (Low-Rank Representation) for parameter reconciliation. Default is False.
    r : int, optional
        The rank for the LoRR parameter reconciliation. Default is 3.
    with_residual : bool, optional
        Whether to enable residual connections in the layers. Default is False.
    channel_num : int, optional
        The number of channels in each layer. Default is 1.
    width : int, optional
        The number of parallel heads in each layer. Default is 1.
    device : str, optional
        The device to use for computation ('cpu' or 'cuda'). Default is 'cpu'.
    *args : optional
        Additional positional arguments for the `rpn` superclass.
    **kwargs : optional
        Additional keyword arguments for the `rpn` superclass.

    Raises
    ------
    ValueError
        If `dims` contains fewer than two dimensions.

    Methods
    -------
    __init__(dims, name='rpn_mlp', enable_bias=False, ...)
        Initializes the MLP model and builds its layers.
    """
    def __init__(
        self,
        dims: list[int] | tuple[int],
        name: str = 'rpn_mlp',
        enable_bias: bool = False,
        # optional parameters
        with_taylor: bool = False,
        d: int = 2,
        with_lorr: bool = False,
        r: int = 3,
        with_residual: bool = False,
        channel_num: int = 1,
        width: int = 1,
        # other parameters
        device: str = 'cpu', *args, **kwargs
    ):
        """
        Initializes the MLP model.

        Parameters
        ----------
        dims : list[int] | tuple[int]
            A list or tuple of integers representing the dimensions of each layer in the MLP.
            Must contain at least two dimensions.
        name : str, optional
            The name of the MLP model. Default is 'rpn_mlp'.
        enable_bias : bool, optional
            Whether to enable bias in the layers. Default is False.
        with_taylor : bool, optional
            Whether to use Taylor expansion for data transformation. Default is False.
        d : int, optional
            The degree of the Taylor expansion if `with_taylor` is True. Default is 2.
        with_lorr : bool, optional
            Whether to use LoRR (Low-Rank Representation) for parameter reconciliation. Default is False.
        r : int, optional
            The rank for the LoRR parameter reconciliation. Default is 3.
        with_residual : bool, optional
            Whether to enable residual connections in the layers. Default is False.
        channel_num : int, optional
            The number of channels in each layer. Default is 1.
        width : int, optional
            The number of parallel heads in each layer. Default is 1.
        device : str, optional
            The device to use for computation ('cpu' or 'cuda'). Default is 'cpu'.
        *args : optional
            Additional positional arguments for the `rpn` superclass.
        **kwargs : optional
            Additional keyword arguments for the `rpn` superclass.

        Raises
        ------
        ValueError
            If `dims` contains fewer than two dimensions.
        """
        if len(dims) < 2:
            raise ValueError("At least two dim values is needed for defining the model...")

        layers = []
        for m, n in zip(dims, dims[1:]):
            layers.append(
                perceptron_layer(
                    m=m, n=n, device=device,
                    enable_bias=enable_bias,
                    with_taylor=with_taylor,
                    with_lorr=with_lorr,
                    with_residual=with_residual,
                    d=d, r=r,
                    channel_num=channel_num,
                    width=width,
                )
            )
        super().__init__(name=name, layers=layers, device=device, *args, **kwargs)

__init__(dims, name='rpn_mlp', enable_bias=False, with_taylor=False, d=2, with_lorr=False, r=3, with_residual=False, channel_num=1, width=1, device='cpu', *args, **kwargs)

Initializes the MLP model.

Parameters:

Name Type Description Default
dims list[int] | tuple[int]

A list or tuple of integers representing the dimensions of each layer in the MLP. Must contain at least two dimensions.

 required
name str

The name of the MLP model. Default is 'rpn_mlp'.

 'rpn_mlp'
enable_bias bool

Whether to enable bias in the layers. Default is False.

 False
with_taylor bool

Whether to use Taylor expansion for data transformation. Default is False.

 False
d int

The degree of the Taylor expansion if with_taylor is True. Default is 2.

 2
with_lorr bool

Whether to use LoRR (Low-Rank Representation) for parameter reconciliation. Default is False.

 False
r int

The rank for the LoRR parameter reconciliation. Default is 3.

 3
with_residual bool

Whether to enable residual connections in the layers. Default is False.

 False
channel_num int

The number of channels in each layer. Default is 1.

 1
width int

The number of parallel heads in each layer. Default is 1.

 1
device str

The device to use for computation ('cpu' or 'cuda'). Default is 'cpu'.

 'cpu'
*args optional

Additional positional arguments for the rpn superclass.

 ()
**kwargs optional

Additional keyword arguments for the rpn superclass.

 {}

Raises:

Type Description
ValueError

If dims contains fewer than two dimensions.
Source code in tinybig/model/rpn_mlp.py 
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
def __init__(
    self,
    dims: list[int] | tuple[int],
    name: str = 'rpn_mlp',
    enable_bias: bool = False,
    # optional parameters
    with_taylor: bool = False,
    d: int = 2,
    with_lorr: bool = False,
    r: int = 3,
    with_residual: bool = False,
    channel_num: int = 1,
    width: int = 1,
    # other parameters
    device: str = 'cpu', *args, **kwargs
):
    """
    Initializes the MLP model.

    Parameters
    ----------
    dims : list[int] | tuple[int]
        A list or tuple of integers representing the dimensions of each layer in the MLP.
        Must contain at least two dimensions.
    name : str, optional
        The name of the MLP model. Default is 'rpn_mlp'.
    enable_bias : bool, optional
        Whether to enable bias in the layers. Default is False.
    with_taylor : bool, optional
        Whether to use Taylor expansion for data transformation. Default is False.
    d : int, optional
        The degree of the Taylor expansion if `with_taylor` is True. Default is 2.
    with_lorr : bool, optional
        Whether to use LoRR (Low-Rank Representation) for parameter reconciliation. Default is False.
    r : int, optional
        The rank for the LoRR parameter reconciliation. Default is 3.
    with_residual : bool, optional
        Whether to enable residual connections in the layers. Default is False.
    channel_num : int, optional
        The number of channels in each layer. Default is 1.
    width : int, optional
        The number of parallel heads in each layer. Default is 1.
    device : str, optional
        The device to use for computation ('cpu' or 'cuda'). Default is 'cpu'.
    *args : optional
        Additional positional arguments for the `rpn` superclass.
    **kwargs : optional
        Additional keyword arguments for the `rpn` superclass.

    Raises
    ------
    ValueError
        If `dims` contains fewer than two dimensions.
    """
    if len(dims) < 2:
        raise ValueError("At least two dim values is needed for defining the model...")

    layers = []
    for m, n in zip(dims, dims[1:]):
        layers.append(
            perceptron_layer(
                m=m, n=n, device=device,
                enable_bias=enable_bias,
                with_taylor=with_taylor,
                with_lorr=with_lorr,
                with_residual=with_residual,
                d=d, r=r,
                channel_num=channel_num,
                width=width,
            )
        )
    super().__init__(name=name, layers=layers, device=device, *args, **kwargs)