combinatorial_normal_expansion

Bases: combinatorial_expansion

The combinatorial normal data expansion function.

It performs the combinatorial normal probabilistic expansion of the input vector, and returns the expansion result. The class inherits from the base expansion class (i.e., the transformation class in the module directory).

...

Notes

For input vector \(\mathbf{x} \in R^m\), its combinatorial normal probabilistic expansion can be represented as follows: $$ \begin{equation} kappa(\mathbf{x} | \boldsymbol{\theta}) = \left[ \log P\left({\mathbf{x} \choose 1} | \theta_1\right), \log P\left({\mathbf{x} \choose 2} | \theta_2\right), \cdots, \log P\left({\mathbf{x} \choose d} | \theta_d\right) \right] \in {R}^D \end{equation} $$ where term \(P\left({{x}} | \theta_d\right)\) in the above expansion denotes the probability density function of the multivariate normal distribution with hyper-parameter \(\theta_d\), $$ \begin{equation} P\left(x | \theta_d\right) \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma}), \end{equation} $$ where the hyper-parameter \(\theta_d = (\mathbf{\mu}, \mathbf{\Sigma})\) covers the mean vector \(\mathbf{\mu}\) and variance matrix \(\mathbf{\Sigma}\).

For combinatorial normal probabilistic expansion, its output expansion dimensions will be \(D = \sum_{i=1}^d {m \choose i}\), where \(d\) denotes the combinatorial expansion order parameter.

By default, the input and output can also be processed with the optional pre- or post-processing functions in the gaussian rbf expansion function.

Attributes:

Name	Type	Description
name	str, default = 'combinatorial_normal_expansion'	The name of the combinatorial normal expansion function.
d	int, default = 2	The combinatorial expansion order.
with_replacement	bool, default = False	The with_replacement tag for the random combination.

Methods:

Name	Description
__init__	It performs the initialization of the expansion function.
calculate_D	It calculates the expansion space dimension D based on the input dimension parameter m.
forward	It implements the abstract forward method declared in the base expansion class.

Source code in tinybig/expansion/combinatorial_expansion.py

class combinatorial_normal_expansion(combinatorial_expansion):
    r"""
    The combinatorial normal data expansion function.

    It performs the combinatorial normal probabilistic expansion of the input vector, and returns the expansion result.
    The class inherits from the base expansion class (i.e., the transformation class in the module directory).

    ...

    Notes
    ----------
    For input vector $\mathbf{x} \in R^m$, its combinatorial normal probabilistic expansion can be represented as follows:
    $$
    \begin{equation}
        kappa(\mathbf{x} | \boldsymbol{\theta}) = \left[ \log P\left({\mathbf{x} \choose 1} | \theta_1\right), \log P\left({\mathbf{x} \choose 2} | \theta_2\right), \cdots, \log P\left({\mathbf{x} \choose d} | \theta_d\right)  \right] \in {R}^D
    \end{equation}
    $$
    where term $P\left({{x}} | \theta_d\right)$ in the above expansion denotes the probability density function of the multivariate normal distribution with hyper-parameter $\theta_d$,
    $$
        \begin{equation}
            P\left(x | \theta_d\right) \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma}),
        \end{equation}
    $$
    where the hyper-parameter $\theta_d = (\mathbf{\mu}, \mathbf{\Sigma})$ covers the mean vector $\mathbf{\mu}$ and variance matrix $\mathbf{\Sigma}$.

    For combinatorial normal probabilistic expansion, its output expansion dimensions will be $D = \sum_{i=1}^d {m \choose i}$,
    where $d$ denotes the combinatorial expansion order parameter.

    By default, the input and output can also be processed with the optional pre- or post-processing functions
    in the gaussian rbf expansion function.

    Attributes
    ----------
    name: str, default = 'combinatorial_normal_expansion'
        The name of the combinatorial normal expansion function.
    d: int, default = 2
        The combinatorial expansion order.
    with_replacement: bool, default = False
        The with_replacement tag for the random combination.

    Methods
    ----------
    __init__
        It performs the initialization of the expansion function.

    calculate_D
        It calculates the expansion space dimension D based on the input dimension parameter m.

    forward
        It implements the abstract forward method declared in the base expansion class.

    """
    def __init__(self, name: str = 'combinatorial_normal_expansion', d: int = 1, with_replacement: bool = False, *args, **kwargs):
        r"""
        The initialization method of the combinatorial normal probabilistic expansion function.

        It initializes a combinatorial normal probabilistic expansion object based on the input function name.
        This method will also call the initialization method of the base class as well.

        Parameters
        ----------
        name: str, default = 'combinatorial_normal_expansion'
            The name of the combinatorial normal probabilistic expansion function.
        d: int, default = 2
            The order of random combinations.
        with_replacement: bool, default = False
            The replacement boolean tag.

        Returns
        ----------
        transformation
            The combinatorial normal probabilistic expansion function.
        """
        print('combinatorial_normal_expansion initialization')
        super().__init__(name=name, d=d, with_replacement=with_replacement, *args, **kwargs)

    def calculate_D(self, m: int):
        r"""
        The expansion dimension calculation method.

        It calculates the intermediate expansion space dimension based on the input dimension parameter m.
        For the combinatorial expansion function, the expansion space dimension will be
        $$ D = \sum_{i=1}^d {m \choose i}. $$

        Parameters
        ----------
        m: int
            The dimension of the input space.

        Returns
        -------
        int
            The dimension of the expansion space.
        """
        assert type(self.d) is int and self.d >= 1
        return int(sum([comb(m, r) for r in range(1, self.d+1)]))

    def forward(self, x: torch.Tensor, device='cpu', with_replacement: bool = False, *args, **kwargs):
        r"""
        The forward method of the combinatorial normal probabilistic expansion function.

        It performs the combinatorial data expansion of the input data and returns the expansion result as
        $$
        \begin{equation}
            \kappa(\mathbf{x}) = \left[ {\mathbf{x} \choose 1}, {\mathbf{x} \choose 2}, \cdots, {\mathbf{x} \choose d} \right] \in {R}^D.
        \end{equation}
        $$


        Parameters
        ----------
        x: torch.Tensor
            The input data vector.
        device: str, default = 'cpu'
            The device to perform the data expansion.
        with_replacement: bool, default = False
            The replacement boolean tag.

        Returns
        ----------
        torch.Tensor
            The expanded data vector of the input.
        """
        b, m = x.shape
        x = self.pre_process(x=x, device=device)

        expansion_shape = list(x.size())
        expansion_shape[-1] = self.calculate_D(m=expansion_shape[-1])

        if len(x.shape) == 1:
            x = torch.unsqueeze(x, 0)
        if len(x.shape) >= 3:
            raise ValueError("Input x can only be 2d or 1d, higher dimensional inputs are not supported yet...")

        x = x.to('cpu')
        with_replacement = with_replacement if with_replacement is not None else self.with_replacement
        combinations = self.combinatorial(x=x, d=self.d, device=device, with_replacement=with_replacement, *args, **kwargs)

        result = torch.zeros(x.size(0), self.calculate_D(m=x.size(1)))
        distribution_dict = {}
        current_index = 0
        for r in range(1, self.d+1):
            if r not in distribution_dict:
                # multivariate_normal_distributions
                distribution_dict[r] = torch.distributions.multivariate_normal.MultivariateNormal(
                    loc=torch.zeros(r), covariance_matrix=torch.eye(r)
                )
            degree_batch_expansion = torch.stack(combinations[r-1], dim=0)
            tuple_count = len(degree_batch_expansion[0])
            degree_batch_log_likelihood = distribution_dict[r].log_prob(value=degree_batch_expansion)
            result[:, current_index:current_index+tuple_count] = degree_batch_log_likelihood
            current_index += tuple_count

        expansion = result.view(*expansion_shape)
        assert expansion.shape == (b, self.calculate_D(m=m))
        return self.post_process(x=expansion, device=device).to(device)

__init__(name='combinatorial_normal_expansion', d=1, with_replacement=False, *args, **kwargs)

The initialization method of the combinatorial normal probabilistic expansion function.

It initializes a combinatorial normal probabilistic expansion object based on the input function name. This method will also call the initialization method of the base class as well.

Parameters:

Name	Type	Description	Default
name	str	The name of the combinatorial normal probabilistic expansion function.	'combinatorial_normal_expansion'
d	int	The order of random combinations.	1
with_replacement	bool	The replacement boolean tag.	False

Returns:

Type	Description
transformation	The combinatorial normal probabilistic expansion function.

Source code in tinybig/expansion/combinatorial_expansion.py

def __init__(self, name: str = 'combinatorial_normal_expansion', d: int = 1, with_replacement: bool = False, *args, **kwargs):
    r"""
    The initialization method of the combinatorial normal probabilistic expansion function.

    It initializes a combinatorial normal probabilistic expansion object based on the input function name.
    This method will also call the initialization method of the base class as well.

    Parameters
    ----------
    name: str, default = 'combinatorial_normal_expansion'
        The name of the combinatorial normal probabilistic expansion function.
    d: int, default = 2
        The order of random combinations.
    with_replacement: bool, default = False
        The replacement boolean tag.

    Returns
    ----------
    transformation
        The combinatorial normal probabilistic expansion function.
    """
    print('combinatorial_normal_expansion initialization')
    super().__init__(name=name, d=d, with_replacement=with_replacement, *args, **kwargs)

calculate_D(m)

The expansion dimension calculation method.

It calculates the intermediate expansion space dimension based on the input dimension parameter m. For the combinatorial expansion function, the expansion space dimension will be $$ D = \sum_{i=1}^d {m \choose i}. $$

Parameters:

Name	Type	Description	Default
m	int	The dimension of the input space.	required

Returns:

Type	Description
int	The dimension of the expansion space.

Source code in tinybig/expansion/combinatorial_expansion.py

def calculate_D(self, m: int):
    r"""
    The expansion dimension calculation method.

    It calculates the intermediate expansion space dimension based on the input dimension parameter m.
    For the combinatorial expansion function, the expansion space dimension will be
    $$ D = \sum_{i=1}^d {m \choose i}. $$

    Parameters
    ----------
    m: int
        The dimension of the input space.

    Returns
    -------
    int
        The dimension of the expansion space.
    """
    assert type(self.d) is int and self.d >= 1
    return int(sum([comb(m, r) for r in range(1, self.d+1)]))

forward(x, device='cpu', with_replacement=False, *args, **kwargs)

The forward method of the combinatorial normal probabilistic expansion function.

It performs the combinatorial data expansion of the input data and returns the expansion result as $$ \begin{equation} \kappa(\mathbf{x}) = \left[ {\mathbf{x} \choose 1}, {\mathbf{x} \choose 2}, \cdots, {\mathbf{x} \choose d} \right] \in {R}^D. \end{equation} $$

Parameters:

Name	Type	Description	Default
x	Tensor	The input data vector.	required
device		The device to perform the data expansion.	'cpu'
with_replacement	bool	The replacement boolean tag.	False

Returns:

Type	Description
Tensor	The expanded data vector of the input.

Source code in tinybig/expansion/combinatorial_expansion.py

def forward(self, x: torch.Tensor, device='cpu', with_replacement: bool = False, *args, **kwargs):
    r"""
    The forward method of the combinatorial normal probabilistic expansion function.

    It performs the combinatorial data expansion of the input data and returns the expansion result as
    $$
    \begin{equation}
        \kappa(\mathbf{x}) = \left[ {\mathbf{x} \choose 1}, {\mathbf{x} \choose 2}, \cdots, {\mathbf{x} \choose d} \right] \in {R}^D.
    \end{equation}
    $$


    Parameters
    ----------
    x: torch.Tensor
        The input data vector.
    device: str, default = 'cpu'
        The device to perform the data expansion.
    with_replacement: bool, default = False
        The replacement boolean tag.

    Returns
    ----------
    torch.Tensor
        The expanded data vector of the input.
    """
    b, m = x.shape
    x = self.pre_process(x=x, device=device)

    expansion_shape = list(x.size())
    expansion_shape[-1] = self.calculate_D(m=expansion_shape[-1])

    if len(x.shape) == 1:
        x = torch.unsqueeze(x, 0)
    if len(x.shape) >= 3:
        raise ValueError("Input x can only be 2d or 1d, higher dimensional inputs are not supported yet...")

    x = x.to('cpu')
    with_replacement = with_replacement if with_replacement is not None else self.with_replacement
    combinations = self.combinatorial(x=x, d=self.d, device=device, with_replacement=with_replacement, *args, **kwargs)

    result = torch.zeros(x.size(0), self.calculate_D(m=x.size(1)))
    distribution_dict = {}
    current_index = 0
    for r in range(1, self.d+1):
        if r not in distribution_dict:
            # multivariate_normal_distributions
            distribution_dict[r] = torch.distributions.multivariate_normal.MultivariateNormal(
                loc=torch.zeros(r), covariance_matrix=torch.eye(r)
            )
        degree_batch_expansion = torch.stack(combinations[r-1], dim=0)
        tuple_count = len(degree_batch_expansion[0])
        degree_batch_log_likelihood = distribution_dict[r].log_prob(value=degree_batch_expansion)
        result[:, current_index:current_index+tuple_count] = degree_batch_log_likelihood
        current_index += tuple_count

    expansion = result.view(*expansion_shape)
    assert expansion.shape == (b, self.calculate_D(m=m))
    return self.post_process(x=expansion, device=device).to(device)