Skip to content

rpn

Bases: model

The RPN model build in the tinyBIG toolkit.

It implements the RPN model build with a deep architecture involving multiple RPN layers.

Notes

The RPN model can be built with a deep architecture by stacking multiple RPN layers on top of each other. A deep RPN model with K layers can be represented as follows: $$ \begin{equation} \begin{cases} \text{Input: } & \mathbf{h}_0 = \mathbf{x},\\ \text{Layer 1: } & \mathbf{h}_1 = \left\langle \kappa_1(\mathbf{h}_0), \psi_1(\mathbf{w}_1) \right\rangle + \pi_1(\mathbf{h}_0),\\ \text{Layer 2: } & \mathbf{h}_2 = \left\langle \kappa_2(\mathbf{h}_1), \psi_2(\mathbf{w}_2) \right\rangle + \pi_2(\mathbf{h}_1),\\ \cdots & \cdots \ \cdots\\ \text{Layer K: } & \mathbf{h}_K = \left\langle \kappa_K(\mathbf{h}_{K-1}), \psi_K(\mathbf{w}_K) \right\rangle + \pi_K(\mathbf{h}_{K-1}),\\ \text{Output: } & \hat{\mathbf{y}} = \mathbf{h}_K. \end{cases} \end{equation} $$ Each of the layers shown above can be implemented with one RPN layer, which may also involve multi-heads and multi-channels.

Attributes:

Name Type Description
name str, default = 'Reconciled_Polynomial_Network'

Name of the RPN model.
layers list, default = None

The model architecture with multiple layers.
device str, default = 'cpu'

Device to host the RPN model.

Methods:

Name Description
__init__

The RPN model initialization method, it will initialize the model architecture based on the input configurations.
get_depthber

The RPN model depth retrieval method, it will get the number of layers involved in the RPN model.
initialize_parameters

The RPN model parameter initialization method, it will initialize the parameters of each layer.
forward

The forward method of the RPN model. It will generate the outputs based on the input data.
Source code in tinybig/model/rpn.py 
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
class rpn(model):
    r"""
    The RPN model build in the tinyBIG toolkit.

    It implements the RPN model build with a deep architecture involving multiple RPN layers.

    Notes
    ----------
    The RPN model can be built with a deep architecture by stacking multiple RPN layers on top of each other.
    A deep RPN model with K layers can be represented as follows:
    $$
        \begin{equation}
            \begin{cases}
                \text{Input: } & \mathbf{h}_0  = \mathbf{x},\\\\
                \text{Layer 1: } & \mathbf{h}\_1 = \left\langle \kappa\_1(\mathbf{h}\_0), \psi\_1(\mathbf{w}\_1) \right\rangle + \pi\_1(\mathbf{h}\_0),\\\\
                \text{Layer 2: } & \mathbf{h}\_2 = \left\langle \kappa\_2(\mathbf{h}\_1), \psi\_2(\mathbf{w}\_2) \right\rangle + \pi\_2(\mathbf{h}\_1),\\\\
                \cdots & \cdots \ \cdots\\\\
                \text{Layer K: } & \mathbf{h}\_K = \left\langle \kappa\_K(\mathbf{h}\_{K-1}), \psi\_K(\mathbf{w}\_K) \right\rangle + \pi\_K(\mathbf{h}\_{K-1}),\\\\
                \text{Output: } & \hat{\mathbf{y}}  = \mathbf{h}_K.
            \end{cases}
        \end{equation}
    $$
    Each of the layers shown above can be implemented with one RPN layer, which may also involve multi-heads and multi-channels.

    Attributes
    ----------
    name: str, default = 'Reconciled_Polynomial_Network'
        Name of the RPN model.
    layers: list, default = None
        The model architecture with multiple layers.
    device: str, default = 'cpu'
        Device to host the RPN model.

    Methods
    ----------
    __init__
        The RPN model initialization method, it will initialize the model architecture based on the input configurations.
    get_depthber
        The RPN model depth retrieval method, it will get the number of layers involved in the RPN model.
    initialize_parameters
        The RPN model parameter initialization method, it will initialize the parameters of each layer.
    forward
        The forward method of the RPN model. It will generate the outputs based on the input data.
    """
    def __init__(
        self,
        name='Reconciled_Polynomial_Network',
        layers: list = None,
        depth: int = None,
        depth_alloc: int | list = None,
        layer_configs: dict | list = None,
        device: str = 'cpu', *args, **kwargs
    ):
        """
        The initialization method of the RPN model with multiple RPN layers.

        It initializes the deep RPN model composed with multiple RPN layers.
        Specifically, this method initializes the name, layers and devices to host the RPN model.

        Parameters
        ----------
        name: str, default = 'Reconciled_Polynomial_Network'
            Name of the RPN model.
        layers: list, default = None
            The list of RPN layers in the model. The layers involved in the model can be initialized
            either directly with the layers parameter or via the layer_configs parameter.
        layer_configs: list, default = None
            The list of RPN layer detailed configurations.
        depth: int, default = None
            The total layer number of the model. It is optional, if the "layers" or the "layer_configs" can provide
            sufficient information for the model initialization, this depth parameter can be set as None.
        depth_alloc: list, default = None
            RPN allows the layers with different configurations, instead of listing such configurations one by one,
            it also allows the listing of each configuration types together with the repeating numbers for
            each of the layers, which are specified by this optional layer allocation parameter.
        device: str, default = 'cpu'
            The device for hosting the RPN layer.

        Returns
        ----------
        object
            The initialized RPN model object.
        """
        # initialize the base model class with the name
        super().__init__(name=name, device=device, *args, **kwargs)

        self.layers = torch.nn.ModuleList()

        if layers is not None:
            self.layers.extend(layers)
            depth = len(self.layers)
        elif layer_configs is not None:
            depth, depth_alloc, layer_configs = model_config.process_num_alloc_configs(depth, depth_alloc, layer_configs)
            assert len(depth_alloc) == len(layer_configs) and depth == sum(depth_alloc)

            for layer_repeat_time, layer_config in zip(depth_alloc, layer_configs):
                for layer_id in range(layer_repeat_time):
                    class_name = layer_config['layer_class']
                    parameters = layer_config['layer_parameters'] if 'layer_parameters' in layer_config else {}
                    parameters['device'] = device
                    layer = model_config.get_obj_from_str(class_name)(**parameters)
                    self.layers.append(layer)
        else:
            raise ValueError("Both layers and layer_configs are None, the model cannot be initialized...")

        assert len(self.layers) == depth
        self.initialize_parameters()

    def get_depth(self):
        """
        RPN model name retrieval method.

        It returns the name of the RPN model.

        Returns
        -------
        str
            It returns the name of the RPN model.
        """
        return len(self.layers)

    def initialize_parameters(self, init_type='xavier_uniform', init_bias=True):
        """
        RPN model parameter initialization method.

        It initializes the parameters of the RPN model with deep architectures.
        This method will call the "initialize_parameters" method for each of the involved layers.

        Returns
        -------
        None
            This method doesn't have any return values.
        """
        for layer in self.layers:
            layer.initialize_parameters(init_type=init_type, init_bias=init_bias)

    def to_config(self):
        model_class = f"tinybig.model.{self.__class__.__name__}"
        model_parameters = {
            'name': self.name,
            'device': self.device,
            'layer_configs': [layer.to_config() for layer in self.layers] if self.layers else [],
        }

        return {
            "model_class": model_class,
            "model_parameters": model_parameters
        }

    def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs):
        r"""
        The forward method of the RPN model.

        It computes the desired outputs based on the data inputs.
        For the RPN with deep architecture involving multiple layers, this method will iteratively call each of the
        layers to process the data inputs, illustrated as follows:

        $$
            \begin{equation}
                \begin{cases}
                    \text{Input: } & \mathbf{h}_0  = \mathbf{x},\\\\
                    \text{Layer 1: } & \mathbf{h}_1 = \left\langle \kappa_1(\mathbf{h}_0), \psi_1(\mathbf{w}_1) \right\rangle + \pi_1(\mathbf{h}_0),\\\\
                    \text{Layer 2: } & \mathbf{h}_2 = \left\langle \kappa_2(\mathbf{h}_1), \psi_2(\mathbf{w}_2) \right\rangle + \pi_2(\mathbf{h}_1),\\\\
                    \cdots & \cdots \ \cdots\\\\
                    \text{Layer K: } & \mathbf{h}_K = \left\langle \kappa_K(\mathbf{h}_{K-1}), \psi_K(\mathbf{w}_K) \right\rangle + \pi_K(\mathbf{h}_{K-1}),\\\\
                    \text{Output: } & \hat{\mathbf{y}}  = \mathbf{h}_K.
                \end{cases}
            \end{equation}
        $$


        Parameters
        ----------
        x: torch.Tensor
            The input data instances.
        device: str, default = 'cpu'
            Device for processing the data inputs.

        Returns
        -------
        torch.Tensor
            The desired outputs generated by the RPN model for the input data instances.
        """
        for layer in self.layers:
            x = layer(x, device=device)
        return x

__init__(name='Reconciled_Polynomial_Network', layers=None, depth=None, depth_alloc=None, layer_configs=None, device='cpu', *args, **kwargs)

The initialization method of the RPN model with multiple RPN layers.

It initializes the deep RPN model composed with multiple RPN layers. Specifically, this method initializes the name, layers and devices to host the RPN model.

Parameters:

Name Type Description Default
name

Name of the RPN model.

 'Reconciled_Polynomial_Network'
layers list

The list of RPN layers in the model. The layers involved in the model can be initialized either directly with the layers parameter or via the layer_configs parameter.

 None
layer_configs dict | list

The list of RPN layer detailed configurations.

 None
depth int

The total layer number of the model. It is optional, if the "layers" or the "layer_configs" can provide sufficient information for the model initialization, this depth parameter can be set as None.

 None
depth_alloc int | list

RPN allows the layers with different configurations, instead of listing such configurations one by one, it also allows the listing of each configuration types together with the repeating numbers for each of the layers, which are specified by this optional layer allocation parameter.

 None
device str

The device for hosting the RPN layer.

 'cpu'

Returns:

Type Description
object

The initialized RPN model object.
Source code in tinybig/model/rpn.py 
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
def __init__(
    self,
    name='Reconciled_Polynomial_Network',
    layers: list = None,
    depth: int = None,
    depth_alloc: int | list = None,
    layer_configs: dict | list = None,
    device: str = 'cpu', *args, **kwargs
):
    """
    The initialization method of the RPN model with multiple RPN layers.

    It initializes the deep RPN model composed with multiple RPN layers.
    Specifically, this method initializes the name, layers and devices to host the RPN model.

    Parameters
    ----------
    name: str, default = 'Reconciled_Polynomial_Network'
        Name of the RPN model.
    layers: list, default = None
        The list of RPN layers in the model. The layers involved in the model can be initialized
        either directly with the layers parameter or via the layer_configs parameter.
    layer_configs: list, default = None
        The list of RPN layer detailed configurations.
    depth: int, default = None
        The total layer number of the model. It is optional, if the "layers" or the "layer_configs" can provide
        sufficient information for the model initialization, this depth parameter can be set as None.
    depth_alloc: list, default = None
        RPN allows the layers with different configurations, instead of listing such configurations one by one,
        it also allows the listing of each configuration types together with the repeating numbers for
        each of the layers, which are specified by this optional layer allocation parameter.
    device: str, default = 'cpu'
        The device for hosting the RPN layer.

    Returns
    ----------
    object
        The initialized RPN model object.
    """
    # initialize the base model class with the name
    super().__init__(name=name, device=device, *args, **kwargs)

    self.layers = torch.nn.ModuleList()

    if layers is not None:
        self.layers.extend(layers)
        depth = len(self.layers)
    elif layer_configs is not None:
        depth, depth_alloc, layer_configs = model_config.process_num_alloc_configs(depth, depth_alloc, layer_configs)
        assert len(depth_alloc) == len(layer_configs) and depth == sum(depth_alloc)

        for layer_repeat_time, layer_config in zip(depth_alloc, layer_configs):
            for layer_id in range(layer_repeat_time):
                class_name = layer_config['layer_class']
                parameters = layer_config['layer_parameters'] if 'layer_parameters' in layer_config else {}
                parameters['device'] = device
                layer = model_config.get_obj_from_str(class_name)(**parameters)
                self.layers.append(layer)
    else:
        raise ValueError("Both layers and layer_configs are None, the model cannot be initialized...")

    assert len(self.layers) == depth
    self.initialize_parameters()

forward(x, device='cpu', *args, **kwargs)

The forward method of the RPN model.

It computes the desired outputs based on the data inputs. For the RPN with deep architecture involving multiple layers, this method will iteratively call each of the layers to process the data inputs, illustrated as follows:

\[
    \begin{equation}
        \begin{cases}
            \text{Input: } & \mathbf{h}_0  = \mathbf{x},\\\\
            \text{Layer 1: } & \mathbf{h}_1 = \left\langle \kappa_1(\mathbf{h}_0), \psi_1(\mathbf{w}_1) \right\rangle + \pi_1(\mathbf{h}_0),\\\\
            \text{Layer 2: } & \mathbf{h}_2 = \left\langle \kappa_2(\mathbf{h}_1), \psi_2(\mathbf{w}_2) \right\rangle + \pi_2(\mathbf{h}_1),\\\\
            \cdots & \cdots \ \cdots\\\\
            \text{Layer K: } & \mathbf{h}_K = \left\langle \kappa_K(\mathbf{h}_{K-1}), \psi_K(\mathbf{w}_K) \right\rangle + \pi_K(\mathbf{h}_{K-1}),\\\\
            \text{Output: } & \hat{\mathbf{y}}  = \mathbf{h}_K.
        \end{cases}
    \end{equation}
\]

Parameters:

Name Type Description Default
x Tensor

The input data instances.

 required
device

Device for processing the data inputs.

 'cpu'

Returns:

Type Description
Tensor

The desired outputs generated by the RPN model for the input data instances.
Source code in tinybig/model/rpn.py 
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs):
    r"""
    The forward method of the RPN model.

    It computes the desired outputs based on the data inputs.
    For the RPN with deep architecture involving multiple layers, this method will iteratively call each of the
    layers to process the data inputs, illustrated as follows:

    $$
        \begin{equation}
            \begin{cases}
                \text{Input: } & \mathbf{h}_0  = \mathbf{x},\\\\
                \text{Layer 1: } & \mathbf{h}_1 = \left\langle \kappa_1(\mathbf{h}_0), \psi_1(\mathbf{w}_1) \right\rangle + \pi_1(\mathbf{h}_0),\\\\
                \text{Layer 2: } & \mathbf{h}_2 = \left\langle \kappa_2(\mathbf{h}_1), \psi_2(\mathbf{w}_2) \right\rangle + \pi_2(\mathbf{h}_1),\\\\
                \cdots & \cdots \ \cdots\\\\
                \text{Layer K: } & \mathbf{h}_K = \left\langle \kappa_K(\mathbf{h}_{K-1}), \psi_K(\mathbf{w}_K) \right\rangle + \pi_K(\mathbf{h}_{K-1}),\\\\
                \text{Output: } & \hat{\mathbf{y}}  = \mathbf{h}_K.
            \end{cases}
        \end{equation}
    $$


    Parameters
    ----------
    x: torch.Tensor
        The input data instances.
    device: str, default = 'cpu'
        Device for processing the data inputs.

    Returns
    -------
    torch.Tensor
        The desired outputs generated by the RPN model for the input data instances.
    """
    for layer in self.layers:
        x = layer(x, device=device)
    return x

get_depth()

RPN model name retrieval method.

It returns the name of the RPN model.

Returns:

Type Description
str

It returns the name of the RPN model.
Source code in tinybig/model/rpn.py 
129
130
131
132
133
134
135
136
137
138
139
140
def get_depth(self):
    """
    RPN model name retrieval method.

    It returns the name of the RPN model.

    Returns
    -------
    str
        It returns the name of the RPN model.
    """
    return len(self.layers)

initialize_parameters(init_type='xavier_uniform', init_bias=True)

RPN model parameter initialization method.

It initializes the parameters of the RPN model with deep architectures. This method will call the "initialize_parameters" method for each of the involved layers.

Returns:

Type Description
None

This method doesn't have any return values.
Source code in tinybig/model/rpn.py 
142
143
144
145
146
147
148
149
150
151
152
153
154
155
def initialize_parameters(self, init_type='xavier_uniform', init_bias=True):
    """
    RPN model parameter initialization method.

    It initializes the parameters of the RPN model with deep architectures.
    This method will call the "initialize_parameters" method for each of the involved layers.

    Returns
    -------
    None
        This method doesn't have any return values.
    """
    for layer in self.layers:
        layer.initialize_parameters(init_type=init_type, init_bias=init_bias)