Skip to content

hermite_expansion

Bases: transformation

The hermite expansion function.

Applies Hermite polynomial expansion to input data.

Notes

Hermite polynomials, first defined by Pierre-Simon Laplace in 1810 and studied in detail by Pafnuty Chebyshev in 1859, were later named after Charles Hermite, who published work on these polynomials in 1864. The Hermite polynomials can be defined in various forms:

Probabilist's Hermite polynomials:

\[
    \begin{equation}
    He_n(x) = (-1)^n \exp \left(\frac{x^2}{2} \right) \frac{\mathrm{d}^n}{\mathrm{d}x^n} \exp \left(- \frac{x^2}{2} \right).
    \end{equation}
\]

Physicist's Hermite polynomials:

\[
    \begin{equation}
    H_n(x) = (-1)^n \exp \left(x^2 \right) \frac{\mathrm{d}^n}{\mathrm{d}x^n} \exp \left(- x^2 \right).
    \end{equation}
\]

These two forms are not identical but can be reduced to each via rescaling:

\[
    \begin{equation}
    H_n(x) = 2^{\frac{n}{2}} He_n (\sqrt{2}x) \text{, and } He_n(x) = 2^{-\frac{n}{2}} H_n \left(\frac{x}{\sqrt{2}} \right).
    \end{equation}
\]

In this paper, we will use the Probabilist's Hermite polynomials for to define the data expansion function by default, which can be formally defined as the following recursive representations:

\[
    \begin{equation}
    He_{n+1}(x)  = x He_n(x) - n He_{n-1}(x), \forall n \ge 1.
    \end{equation}
\]

Some examples of the Probabilist's Hermite polynomials are also illustrated as follows:

\[
    \begin{equation}
    \begin{aligned}
    He_{0}(x) &= 1;\\
    He_{1}(x) &= x;\\
    He_{2}(x) &= x^2 - 1;\\
    He_{3}(x) &= x^3 - 3x;\\
    He_{4}(x) &= x^4 - 6x^2 + 3.\\
    \end{aligned}
    \end{equation}
\]

Based on the Probabilist's Hermite polynomials, we can define the data expansion function with order \(d\) as follows:

\[
    \begin{equation}
    \kappa(\mathbf{x} | d)  = \left[ He_1(\mathbf{x}), He_2(\mathbf{x}), \cdots, He_d(\mathbf{x}) \right] \in R^D,
    \end{equation}
\]

where \(d\) is the order hyper-parameter and the output dimension \(D = md\).

Attributes:

Name Type Description
d int

The degree of Hermite polynomial expansion.

Methods:

Name Description
calculate_D

Calculates the output dimension after expansion.
forward

Performs Hermite polynomial expansion on the input tensor.
Source code in tinybig/expansion/orthogonal_polynomial_expansion.py 
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
class hermite_expansion(transformation):
    r"""
        The hermite expansion function.

        Applies Hermite polynomial expansion to input data.

        Notes
        ---------
        Hermite polynomials, first defined by Pierre-Simon Laplace in 1810 and studied in detail by Pafnuty Chebyshev in 1859, were later named after Charles Hermite, who published work on these polynomials in 1864. The Hermite polynomials can be defined in various forms:

        __Probabilist's Hermite polynomials:__

        $$
            \begin{equation}
            He_n(x) = (-1)^n \exp \left(\frac{x^2}{2} \right) \frac{\mathrm{d}^n}{\mathrm{d}x^n} \exp \left(- \frac{x^2}{2} \right).
            \end{equation}
        $$

        __Physicist's Hermite polynomials:__

        $$
            \begin{equation}
            H_n(x) = (-1)^n \exp \left(x^2 \right) \frac{\mathrm{d}^n}{\mathrm{d}x^n} \exp \left(- x^2 \right).
            \end{equation}
        $$

        These two forms are not identical but can be reduced to each via rescaling:

        $$
            \begin{equation}
            H_n(x) = 2^{\frac{n}{2}} He_n (\sqrt{2}x) \text{, and } He_n(x) = 2^{-\frac{n}{2}} H_n \left(\frac{x}{\sqrt{2}} \right).
            \end{equation}
        $$

        In this paper, we will use the Probabilist's Hermite polynomials for to define the data expansion function by default, which can be formally defined as the following recursive representations:

        $$
            \begin{equation}
            He_{n+1}(x)  = x He_n(x) - n He_{n-1}(x), \forall n \ge 1.
            \end{equation}
        $$

        Some examples of the Probabilist's Hermite polynomials are also illustrated as follows:

        $$
            \begin{equation}
            \begin{aligned}
            He_{0}(x) &= 1;\\
            He_{1}(x) &= x;\\
            He_{2}(x) &= x^2 - 1;\\
            He_{3}(x) &= x^3 - 3x;\\
            He_{4}(x) &= x^4 - 6x^2 + 3.\\
            \end{aligned}
            \end{equation}
        $$

        Based on the Probabilist's Hermite polynomials, we can define the data expansion function with order $d$ as follows:

        $$
            \begin{equation}
            \kappa(\mathbf{x} | d)  = \left[ He_1(\mathbf{x}), He_2(\mathbf{x}), \cdots, He_d(\mathbf{x}) \right] \in R^D,
            \end{equation}
        $$

        where $d$ is the order hyper-parameter and the output dimension $D = md$.

        Attributes
        ----------
        d : int
            The degree of Hermite polynomial expansion.

        Methods
        -------
        calculate_D(m: int)
            Calculates the output dimension after expansion.
        forward(x: torch.Tensor, device='cpu', *args, **kwargs)
            Performs Hermite polynomial expansion on the input tensor.
    """

    def __init__(self, name: str = 'chebyshev_polynomial_expansion', d: int = 2, *args, **kwargs):
        """
            Initializes the Hermite expansion.

            Parameters
            ----------
            name : str, optional
                Name of the expansion. Defaults to 'chebyshev_polynomial_expansion'.
            d : int, optional
                Degree of Hermite polynomial expansion. Defaults to 2.
            *args : tuple
                Additional positional arguments.
            **kwargs : dict
                Additional keyword arguments.
        """
        super().__init__(name=name, *args, **kwargs)
        self.d = d

    def calculate_D(self, m: int):
        """
            Calculates the output dimension after Hermite polynomial expansion.

            Parameters
            ----------
            m : int
                Input dimension.

            Returns
            -------
            int
                Output dimension.
        """
        return m * self.d

    def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs):
        """
            Performs Hermite polynomial expansion on the input tensor.

            Parameters
            ----------
            x : torch.Tensor
                Input tensor of shape `(batch_size, input_dim)`.
            device : str, optional
                Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
            *args : tuple
                Additional positional arguments.
            **kwargs : dict
                Additional keyword arguments.

            Returns
            -------
            torch.Tensor
                Expanded tensor.

            Raises
            ------
            AssertionError
                If the output tensor shape does not match the expected dimensions.
        """
        b, m = x.shape
        x = self.pre_process(x=x, device=device)

        # base case: order 0
        expansion = torch.ones(size=[x.size(0), x.size(1), self.d + 1]).to(device)
        # base case: order 1
        if self.d > 0:
            expansion[:, :, 1] = x
        # high-order cases
        for n in range(2, self.d + 1):
            expansion[:, :, n] = x * expansion[:, :, n-1].clone() - (n-1) * expansion[:, :, n-2].clone()

        expansion = expansion[:, :, 1:].contiguous().view(x.size(0), -1)

        assert expansion.shape == (b, self.calculate_D(m=m))
        return self.post_process(x=expansion, device=device)

__init__(name='chebyshev_polynomial_expansion', d=2, *args, **kwargs)

Initializes the Hermite expansion.

Parameters:

Name Type Description Default
name str

Name of the expansion. Defaults to 'chebyshev_polynomial_expansion'.

 'chebyshev_polynomial_expansion'
d int

Degree of Hermite polynomial expansion. Defaults to 2.

 2
*args tuple

Additional positional arguments.

 ()
**kwargs dict

Additional keyword arguments.

 {}
Source code in tinybig/expansion/orthogonal_polynomial_expansion.py 
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
def __init__(self, name: str = 'chebyshev_polynomial_expansion', d: int = 2, *args, **kwargs):
    """
        Initializes the Hermite expansion.

        Parameters
        ----------
        name : str, optional
            Name of the expansion. Defaults to 'chebyshev_polynomial_expansion'.
        d : int, optional
            Degree of Hermite polynomial expansion. Defaults to 2.
        *args : tuple
            Additional positional arguments.
        **kwargs : dict
            Additional keyword arguments.
    """
    super().__init__(name=name, *args, **kwargs)
    self.d = d

calculate_D(m)

Calculates the output dimension after Hermite polynomial expansion.

Parameters:

Name Type Description Default
m int

Input dimension.

 required

Returns:

Type Description
int

Output dimension.
Source code in tinybig/expansion/orthogonal_polynomial_expansion.py 
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
def calculate_D(self, m: int):
    """
        Calculates the output dimension after Hermite polynomial expansion.

        Parameters
        ----------
        m : int
            Input dimension.

        Returns
        -------
        int
            Output dimension.
    """
    return m * self.d

forward(x, device='cpu', *args, **kwargs)

Performs Hermite polynomial expansion on the input tensor.

Parameters:

Name Type Description Default
x Tensor

Input tensor of shape (batch_size, input_dim).

 required
device str

Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.

 'cpu'
*args tuple

Additional positional arguments.

 ()
**kwargs dict

Additional keyword arguments.

 {}

Returns:

Type Description
Tensor

Expanded tensor.

Raises:

Type Description
AssertionError

If the output tensor shape does not match the expected dimensions.
Source code in tinybig/expansion/orthogonal_polynomial_expansion.py 
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs):
    """
        Performs Hermite polynomial expansion on the input tensor.

        Parameters
        ----------
        x : torch.Tensor
            Input tensor of shape `(batch_size, input_dim)`.
        device : str, optional
            Device for computation ('cpu', 'cuda'). Defaults to 'cpu'.
        *args : tuple
            Additional positional arguments.
        **kwargs : dict
            Additional keyword arguments.

        Returns
        -------
        torch.Tensor
            Expanded tensor.

        Raises
        ------
        AssertionError
            If the output tensor shape does not match the expected dimensions.
    """
    b, m = x.shape
    x = self.pre_process(x=x, device=device)

    # base case: order 0
    expansion = torch.ones(size=[x.size(0), x.size(1), self.d + 1]).to(device)
    # base case: order 1
    if self.d > 0:
        expansion[:, :, 1] = x
    # high-order cases
    for n in range(2, self.d + 1):
        expansion[:, :, n] = x * expansion[:, :, n-1].clone() - (n-1) * expansion[:, :, n-2].clone()

    expansion = expansion[:, :, 1:].contiguous().view(x.size(0), -1)

    assert expansion.shape == (b, self.calculate_D(m=m))
    return self.post_process(x=expansion, device=device)