naive_exponential_expansion

Bases: transformation

The naive exponential data expansion function.

It performs the naive exponential probabilistic expansion of the input vector, and returns the expansion result. The class inherits from the base expansion class (i.e., the transformation class in the module directory).

...

Notes

For input vector \(\mathbf{x} \in R^m\), its naive exponential probabilistic expansion can be represented as follows: $$ \begin{equation} \kappa(\mathbf{x} | \boldsymbol{\theta}) = \left[ \log P\left({\mathbf{x}} | \theta_1\right), \log P\left({\mathbf{x} } | \theta_2\right), \cdots, \log P\left({\mathbf{x} } | \theta_d\right) \right] \in {R}^D \end{equation} $$ where \(P\left({{x}} | \theta_d\right)\) denotes the probability density function of the exponential distribution with hyper-parameter \(\theta_d\), $$ \begin{equation} P\left(x | \theta_d\right) = P(x | \lambda) = \begin{cases} \lambda \exp^{- \lambda x} & \text{ for } x \ge 0,\\ 0 & \text{ otherwise}. \end{cases} \end{equation} $$

For naive exponential probabilistic expansion, its output expansion dimensions will be \(D = md\), where \(d\) denotes the number of provided distribution hyper-parameters.

By default, the input and output can also be processed with the optional pre- or post-processing functions in the gaussian rbf expansion function.

Attributes:

Name	Type	Description
name	str, default = 'naive_exponential_expansion'	Name of the naive exponential expansion function.

Methods:

Name	Description
__init__	It performs the initialization of the expansion function.
calculate_D	It calculates the expansion space dimension D based on the input dimension parameter m.
forward	It implements the abstract forward method declared in the base expansion class.

Source code in tinybig/expansion/probabilistic_expansion.py

class naive_exponential_expansion(transformation):
    r"""
    The naive exponential data expansion function.

    It performs the naive exponential probabilistic expansion of the input vector, and returns the expansion result.
    The class inherits from the base expansion class (i.e., the transformation class in the module directory).

    ...

    Notes
    ----------
    For input vector $\mathbf{x} \in R^m$, its naive exponential probabilistic expansion can be represented as follows:
    $$
    \begin{equation}
        \kappa(\mathbf{x} | \boldsymbol{\theta}) = \left[ \log P\left({\mathbf{x}} | \theta\_1\right), \log P\left({\mathbf{x} } | \theta\_2\right), \cdots, \log P\left({\mathbf{x} } | \theta\_d\right)  \right] \in {R}^D
    \end{equation}
    $$
    where $P\left({{x}} | \theta_d\right)$ denotes the probability density function of the exponential distribution with hyper-parameter $\theta_d$,
    $$
        \begin{equation}
            P\left(x | \theta_d\right) = P(x | \lambda) =   \begin{cases}
                                                                \lambda \exp^{- \lambda x} & \text{ for } x \ge 0,\\\\
                                                                0 & \text{ otherwise}.
                                                            \end{cases}
        \end{equation}
    $$

    For naive exponential probabilistic expansion, its output expansion dimensions will be $D = md$,
    where $d$ denotes the number of provided distribution hyper-parameters.

    By default, the input and output can also be processed with the optional pre- or post-processing functions
    in the gaussian rbf expansion function.

    Attributes
    ----------
    name: str, default = 'naive_exponential_expansion'
        Name of the naive exponential expansion function.

    Methods
    ----------
    __init__
        It performs the initialization of the expansion function.

    calculate_D
        It calculates the expansion space dimension D based on the input dimension parameter m.

    forward
        It implements the abstract forward method declared in the base expansion class.

    """
    def __init__(self, name='naive_exponential_expansion', *args, **kwargs):
        r"""
        The initialization method of the naive exponential probabilistic expansion function.

        It initializes a naive exponential probabilistic expansion object based on the input function name.
        This method will also call the initialization method of the base class as well.

        Parameters
        ----------
        name: str, default = 'naive_exponential_expansion'
            The name of the naive exponential expansion function.

        Returns
        ----------
        transformation
            The naive exponential probabilistic expansion function.
        """
        super().__init__(name=name, *args, **kwargs)

    def calculate_D(self, m: int):
        r"""
        The expansion dimension calculation method.

        It calculates the intermediate expansion space dimension based on the input dimension parameter m.
        For the naive exponential probabilistic expansion function, the expansion space dimension will be
        $$ D = m d, $$
        where $d$ denotes the number of provided distribution hyper-parameters.

        Parameters
        ----------
        m: int
            The dimension of the input space.

        Returns
        -------
        int
            The dimension of the expansion space.
        """
        return m

    def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs):
        r"""
        The forward method of the naive exponential probabilistic expansion function.

        It performs the naive exponential probabilistic expansion of the input data and returns the expansion result as
        $$
        \begin{equation}
            \kappa(\mathbf{x} | \boldsymbol{\theta}) = \left[ \log P\left({\mathbf{x}} | \theta\_1\right), \log P\left({\mathbf{x} } | \theta\_2\right), \cdots, \log P\left({\mathbf{x} } | \theta\_d\right)  \right] \in {R}^D
        \end{equation}
        $$


        Parameters
        ----------
        x: torch.Tensor
            The input data vector.
        device: str, default = 'cpu'
            The device to perform the data expansion.

        Returns
        ----------
        torch.Tensor
            The expanded data vector of the input.
        """
        b, m = x.shape
        x = self.pre_process(x=x, device=device)
        # pre-normalize the input to range [0, 1]
        x = x.to('cpu')
        x = 0.99 * F.sigmoid(x) + 0.001

        exponential_dist_1 = torch.distributions.exponential.Exponential(torch.tensor([0.5]))
        exponential_x_1 = exponential_dist_1.log_prob(x)

        expansion = exponential_x_1

        assert expansion.shape == (b, self.calculate_D(m=m))
        return self.post_process(x=expansion, device=device).to(device)

__init__(name='naive_exponential_expansion', *args, **kwargs)

The initialization method of the naive exponential probabilistic expansion function.

It initializes a naive exponential probabilistic expansion object based on the input function name. This method will also call the initialization method of the base class as well.

Parameters:

Name	Type	Description	Default
name		The name of the naive exponential expansion function.	'naive_exponential_expansion'

Returns:

Type	Description
transformation	The naive exponential probabilistic expansion function.

Source code in tinybig/expansion/probabilistic_expansion.py

def __init__(self, name='naive_exponential_expansion', *args, **kwargs):
    r"""
    The initialization method of the naive exponential probabilistic expansion function.

    It initializes a naive exponential probabilistic expansion object based on the input function name.
    This method will also call the initialization method of the base class as well.

    Parameters
    ----------
    name: str, default = 'naive_exponential_expansion'
        The name of the naive exponential expansion function.

    Returns
    ----------
    transformation
        The naive exponential probabilistic expansion function.
    """
    super().__init__(name=name, *args, **kwargs)

calculate_D(m)

The expansion dimension calculation method.

It calculates the intermediate expansion space dimension based on the input dimension parameter m. For the naive exponential probabilistic expansion function, the expansion space dimension will be $$ D = m d, $$ where \(d\) denotes the number of provided distribution hyper-parameters.

Parameters:

Name	Type	Description	Default
m	int	The dimension of the input space.	required

Returns:

Type	Description
int	The dimension of the expansion space.

Source code in tinybig/expansion/probabilistic_expansion.py

def calculate_D(self, m: int):
    r"""
    The expansion dimension calculation method.

    It calculates the intermediate expansion space dimension based on the input dimension parameter m.
    For the naive exponential probabilistic expansion function, the expansion space dimension will be
    $$ D = m d, $$
    where $d$ denotes the number of provided distribution hyper-parameters.

    Parameters
    ----------
    m: int
        The dimension of the input space.

    Returns
    -------
    int
        The dimension of the expansion space.
    """
    return m

forward(x, device='cpu', *args, **kwargs)

The forward method of the naive exponential probabilistic expansion function.

It performs the naive exponential probabilistic expansion of the input data and returns the expansion result as $$ \begin{equation} \kappa(\mathbf{x} | \boldsymbol{\theta}) = \left[ \log P\left({\mathbf{x}} | \theta_1\right), \log P\left({\mathbf{x} } | \theta_2\right), \cdots, \log P\left({\mathbf{x} } | \theta_d\right) \right] \in {R}^D \end{equation} $$

Parameters:

Name	Type	Description	Default
x	Tensor	The input data vector.	required
device		The device to perform the data expansion.	'cpu'

Returns:

Type	Description
Tensor	The expanded data vector of the input.

Source code in tinybig/expansion/probabilistic_expansion.py

def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs):
    r"""
    The forward method of the naive exponential probabilistic expansion function.

    It performs the naive exponential probabilistic expansion of the input data and returns the expansion result as
    $$
    \begin{equation}
        \kappa(\mathbf{x} | \boldsymbol{\theta}) = \left[ \log P\left({\mathbf{x}} | \theta\_1\right), \log P\left({\mathbf{x} } | \theta\_2\right), \cdots, \log P\left({\mathbf{x} } | \theta\_d\right)  \right] \in {R}^D
    \end{equation}
    $$


    Parameters
    ----------
    x: torch.Tensor
        The input data vector.
    device: str, default = 'cpu'
        The device to perform the data expansion.

    Returns
    ----------
    torch.Tensor
        The expanded data vector of the input.
    """
    b, m = x.shape
    x = self.pre_process(x=x, device=device)
    # pre-normalize the input to range [0, 1]
    x = x.to('cpu')
    x = 0.99 * F.sigmoid(x) + 0.001

    exponential_dist_1 = torch.distributions.exponential.Exponential(torch.tensor([0.5]))
    exponential_x_1 = exponential_dist_1.log_prob(x)

    expansion = exponential_x_1

    assert expansion.shape == (b, self.calculate_D(m=m))
    return self.post_process(x=expansion, device=device).to(device)