Skip to content

identity_expansion

Bases: transformation

The identity data expansion function.

It performs the identity expansion of the input vector, and returns the expansion result. This class inherits from the expansion class (i.e., the transformation class in the module directory).

...

Notes

For the identity expansion function, the expansion space dimension equals to the input space dimension.

For input vector \(\mathbf{x} \in R^m\), its identity expansion will be $$ \begin{equation} \kappa(\mathbf{x}) = \sigma(\mathbf{x}) \in R^D \end{equation} $$ where \(D = m\). By default, we can also process the input with optional pre- or post-processing functions denoted by \(\sigma(\cdot)\) in the above formula.

Attributes:

Name Type Description
name str, default = 'identity_expansion'

Name of the expansion function.

Methods:

Name Description
__init__

It performs the initialization of the expansion function.
calculate_D

It calculates the expansion space dimension D based on the input dimension parameter m.
forward

It implements the abstract forward method declared in the base expansion class.
Source code in tinybig/expansion/basic_expansion.py 
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
class identity_expansion(transformation):
    r"""
    The identity data expansion function.

    It performs the identity expansion of the input vector, and returns the expansion result.
    This class inherits from the expansion class (i.e., the transformation class in the module directory).

    ...

    Notes
    ----------
    For the identity expansion function, the expansion space dimension equals to the input space dimension.

    For input vector $\mathbf{x} \in R^m$, its identity expansion will be
    $$
        \begin{equation}
            \kappa(\mathbf{x}) = \sigma(\mathbf{x}) \in R^D
        \end{equation}
    $$
    where $D = m$. By default, we can also process the input with optional pre- or post-processing functions
    denoted by $\sigma(\cdot)$ in the above formula.

    Attributes
    ----------
    name: str, default = 'identity_expansion'
        Name of the expansion function.

    Methods
    ----------
    __init__
        It performs the initialization of the expansion function.

    calculate_D
        It calculates the expansion space dimension D based on the input dimension parameter m.

    forward
        It implements the abstract forward method declared in the base expansion class.

    """
    def __init__(self, name='identity_expansion', *args, **kwargs):
        """
        The initialization method of the identity expansion function.

        It initializes an identity expansion object based on the input function name.
        This method will also call the initialization method of the base class as well.

        Parameters
        ----------
        name: str, default = 'identity_expansion'
            The name of the identity expansion function.

        Returns
        ----------
        transformation
            The identity expansion function.
        """
        super().__init__(name=name, *args, **kwargs)

    def calculate_D(self, m: int):
        r"""
        The expansion dimension calculation method.

        It calculates the intermediate expansion space dimension based on the input dimension parameter m.
        For the identity expansion function, the expansion space dimension equals to the input space dimension, i.e.,
        $$ D = m. $$

        Parameters
        ----------
        m: int
            The dimension of the input space.

        Returns
        -------
        int
            The dimension of the expansion space.
        """
        return m

    def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs) -> torch.Tensor:
        r"""
        The forward method of the data expansion function.

        It performs the identity data expansion of the input data and returns the expansion result
        according to the following equation:
        $$
            \kappa(\mathbf{x}) = \mathbf{x} \in R^D,
        $$
        with optional pre- and post-processing functions.

        Examples
        ----------
        >>> import torch
        >>> from tinybig.expansion import identity_expansion
        >>> expansion = identity_expansion(name='identity_expansion')
        >>> x = torch.Tensor([0.5, 0.5])
        >>> kappa_x = expansion(x)
        >>> kappa_x
        tensor([0.5000, 0.5000])

        >>> import torch.nn.functional as F
        >>> expansion_with_preprocessing = identity_expansion(name='identity_expansion_with_preprocessing', preprocess_functions=F.relu)
        >>> kappa_x = expansion_with_preprocessing(x)
        >>> kappa_x
        tensor([0.5000, 0.5000])

        >>> expansion_with_postprocessing = identity_expansion(name='identity_expansion_with_postprocessing', postprocess_functions=F.sigmoid)
        >>> kappa_x = expansion_with_postprocessing(x)
        >>> kappa_x
        tensor([0.6225, 0.6225])

        Parameters
        ----------
        x: torch.Tensor
            The input data vector.
        device: str, default = 'cpu'
            The device to perform the data expansion.
        args: list, default = ()
            The other parameters.
        kwargs: dict, default = {}
            The other parameters.

        Returns
        ----------
        torch.Tensor
            The expanded data vector of the input.
        """
        b, m = x.shape
        x = self.pre_process(x=x, device=device)

        expansion = x

        assert expansion.shape == (b, self.calculate_D(m=m))
        return self.post_process(x=expansion, device=device)

__init__(name='identity_expansion', *args, **kwargs)

The initialization method of the identity expansion function.

It initializes an identity expansion object based on the input function name. This method will also call the initialization method of the base class as well.

Parameters:

Name Type Description Default
name

The name of the identity expansion function.

 'identity_expansion'

Returns:

Type Description
transformation

The identity expansion function.
Source code in tinybig/expansion/basic_expansion.py 
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
def __init__(self, name='identity_expansion', *args, **kwargs):
    """
    The initialization method of the identity expansion function.

    It initializes an identity expansion object based on the input function name.
    This method will also call the initialization method of the base class as well.

    Parameters
    ----------
    name: str, default = 'identity_expansion'
        The name of the identity expansion function.

    Returns
    ----------
    transformation
        The identity expansion function.
    """
    super().__init__(name=name, *args, **kwargs)

calculate_D(m)

The expansion dimension calculation method.

It calculates the intermediate expansion space dimension based on the input dimension parameter m. For the identity expansion function, the expansion space dimension equals to the input space dimension, i.e., $$ D = m. $$

Parameters:

Name Type Description Default
m int

The dimension of the input space.

 required

Returns:

Type Description
int

The dimension of the expansion space.
Source code in tinybig/expansion/basic_expansion.py 
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
def calculate_D(self, m: int):
    r"""
    The expansion dimension calculation method.

    It calculates the intermediate expansion space dimension based on the input dimension parameter m.
    For the identity expansion function, the expansion space dimension equals to the input space dimension, i.e.,
    $$ D = m. $$

    Parameters
    ----------
    m: int
        The dimension of the input space.

    Returns
    -------
    int
        The dimension of the expansion space.
    """
    return m

forward(x, device='cpu', *args, **kwargs)

The forward method of the data expansion function.

It performs the identity data expansion of the input data and returns the expansion result according to the following equation: $$ \kappa(\mathbf{x}) = \mathbf{x} \in R^D, $$ with optional pre- and post-processing functions.

Examples:

>>> import torch
>>> from tinybig.expansion import identity_expansion
>>> expansion = identity_expansion(name='identity_expansion')
>>> x = torch.Tensor([0.5, 0.5])
>>> kappa_x = expansion(x)
>>> kappa_x
tensor([0.5000, 0.5000])

>>> import torch.nn.functional as F
>>> expansion_with_preprocessing = identity_expansion(name='identity_expansion_with_preprocessing', preprocess_functions=F.relu)
>>> kappa_x = expansion_with_preprocessing(x)
>>> kappa_x
tensor([0.5000, 0.5000])

>>> expansion_with_postprocessing = identity_expansion(name='identity_expansion_with_postprocessing', postprocess_functions=F.sigmoid)
>>> kappa_x = expansion_with_postprocessing(x)
>>> kappa_x
tensor([0.6225, 0.6225])

Parameters:

Name Type Description Default
x Tensor

The input data vector.

 required
device

The device to perform the data expansion.

 'cpu'
args

The other parameters.

 ()
kwargs

The other parameters.

 {}

Returns:

Type Description
Tensor

The expanded data vector of the input.
Source code in tinybig/expansion/basic_expansion.py 
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs) -> torch.Tensor:
    r"""
    The forward method of the data expansion function.

    It performs the identity data expansion of the input data and returns the expansion result
    according to the following equation:
    $$
        \kappa(\mathbf{x}) = \mathbf{x} \in R^D,
    $$
    with optional pre- and post-processing functions.

    Examples
    ----------
    >>> import torch
    >>> from tinybig.expansion import identity_expansion
    >>> expansion = identity_expansion(name='identity_expansion')
    >>> x = torch.Tensor([0.5, 0.5])
    >>> kappa_x = expansion(x)
    >>> kappa_x
    tensor([0.5000, 0.5000])

    >>> import torch.nn.functional as F
    >>> expansion_with_preprocessing = identity_expansion(name='identity_expansion_with_preprocessing', preprocess_functions=F.relu)
    >>> kappa_x = expansion_with_preprocessing(x)
    >>> kappa_x
    tensor([0.5000, 0.5000])

    >>> expansion_with_postprocessing = identity_expansion(name='identity_expansion_with_postprocessing', postprocess_functions=F.sigmoid)
    >>> kappa_x = expansion_with_postprocessing(x)
    >>> kappa_x
    tensor([0.6225, 0.6225])

    Parameters
    ----------
    x: torch.Tensor
        The input data vector.
    device: str, default = 'cpu'
        The device to perform the data expansion.
    args: list, default = ()
        The other parameters.
    kwargs: dict, default = {}
        The other parameters.

    Returns
    ----------
    torch.Tensor
        The expanded data vector of the input.
    """
    b, m = x.shape
    x = self.pre_process(x=x, device=device)

    expansion = x

    assert expansion.shape == (b, self.calculate_D(m=m))
    return self.post_process(x=expansion, device=device)