Skip to content

metric_compression

Bases: transformation

The metric based compression function.

It performs the data compression based on provided compression metric, which can be min, max, mean, prod, etc.

...

Notes

Formally, given a data instance \(\mathbf{x} \in R^m\) and a provided metric \(\phi: {R}^m \to {R}^{d_{\phi}}\), which transforms it into a dense representation of length \(d_{\phi}\), we can represent the metric based compression function as follows:

\[
    \begin{equation}
    \kappa(\mathbf{x}) = \phi(\mathbf{x}) \in {R}^{d},
    \end{equation}
\]

where the compression output vector dimension is \(d = d_{\phi}\). For the majority of metric \(\phi\) studied in this project, the output is typically a scalar, i.e., the dimension \(d_{\phi} = 1\).

Attributes:

Name Type Description
metric Callable[[Tensor], Tensor]

The metric compression metric.
name str, default = 'metric_compression'

Name of the compression function.

Methods:

Name Description
__init__

It performs the initialization of the metric compression function.
calculate_D

It calculates the compression space dimension d based on the input dimension parameter m.
forward

It implements the abstract forward method to define the compression function.
Source code in tinybig/compression/metric_based_compression.py 
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
class metric_compression(transformation):
    r"""
        The metric based compression function.

        It performs the data compression based on provided compression metric, which can be min, max, mean, prod, etc.

        ...

        Notes
        ----------
        Formally, given a data instance $\mathbf{x} \in R^m$ and a provided metric $\phi: {R}^m \to {R}^{d_{\phi}}$,
        which transforms it into a dense representation of length $d_{\phi}$,
        we can represent the metric based compression function as follows:

        $$
            \begin{equation}
            \kappa(\mathbf{x}) = \phi(\mathbf{x}) \in {R}^{d},
            \end{equation}
        $$

        where the compression output vector dimension is $d = d_{\phi}$. For the majority of metric $\phi$ studied in this project,
        the output is typically a scalar, i.e., the dimension $d_{\phi} = 1$.

        Attributes
        ----------
        metric: Callable[[torch.Tensor], torch.Tensor]
            The metric compression metric.
        name: str, default = 'metric_compression'
            Name of the compression function.

        Methods
        ----------
        __init__
            It performs the initialization of the metric compression function.

        calculate_D
            It calculates the compression space dimension d based on the input dimension parameter m.

        forward
            It implements the abstract forward method to define the compression function.

    """

    def __init__(self, metric: Callable[[torch.Tensor], torch.Tensor], name: str = 'metric_compression', *args, **kwargs):
        """
            The initialization method of the metric based compression function.

            It initializes the metric compression function based on the provided metric mapping.

            Parameters
            ----------
            metric: Callable[[torch.Tensor], torch.Tensor]
                The metric based compression metric.
            name: str, default = 'metric_compression'
                Name of the compression function.

            Returns
            ----------
            transformation
                The metric based compression function.
        """
        super().__init__(name=name, *args, **kwargs)
        self.metric = metric

    def calculate_D(self, m: int):
        r"""
            The metric compression dimension calculation method.

            The compression output vector dimension is $d = d_{\phi}$.
            For the majority of metric $\phi$ studied in this project,
            the output is typically a scalar, i.e., the dimension $d_{\phi} = 1$.

            Parameters
            ----------
            m: int
                The dimension of the input space.

            Returns
            -------
            int
                The dimension of the metric compression space.
        """
        return 1

    def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs):
        r"""
            The forward method of the metric compression function.

            It performs the metric based data compression of the input data and returns the compression result.

            Formally, given a data instance $\mathbf{x} \in R^m$ and a provided metric $\phi: {R}^m \to {R}^{d_{\phi}}$,
            which transforms it into a dense representation of length $d_{\phi}$,
            we can represent the metric based compression function as follows:

            $$
                \begin{equation}
                \kappa(\mathbf{x}) = \phi(\mathbf{x}) \in {R}^{d},
                \end{equation}
            $$

            where the compression output vector dimension is $d = d_{\phi}$.

            Parameters
            ----------
            x: torch.Tensor
                The input data vector.
            device: str, default = 'cpu'
                The device of the input data vector.

            Returns
            -------
            torch.Tensor
                The compression result.
        """
        b, m = x.shape
        x = self.pre_process(x=x, device=device)

        assert self.metric is not None
        compression = self.metric(x).unsqueeze(1)

        assert compression.shape == (b, self.calculate_D(m=m))
        return self.post_process(x=compression, device=device)

__init__(metric, name='metric_compression', *args, **kwargs)

The initialization method of the metric based compression function.

It initializes the metric compression function based on the provided metric mapping.

Parameters:

Name Type Description Default
metric Callable[[Tensor], Tensor]

The metric based compression metric.

 required
name str

Name of the compression function.

 'metric_compression'

Returns:

Type Description
transformation

The metric based compression function.
Source code in tinybig/compression/metric_based_compression.py 
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
def __init__(self, metric: Callable[[torch.Tensor], torch.Tensor], name: str = 'metric_compression', *args, **kwargs):
    """
        The initialization method of the metric based compression function.

        It initializes the metric compression function based on the provided metric mapping.

        Parameters
        ----------
        metric: Callable[[torch.Tensor], torch.Tensor]
            The metric based compression metric.
        name: str, default = 'metric_compression'
            Name of the compression function.

        Returns
        ----------
        transformation
            The metric based compression function.
    """
    super().__init__(name=name, *args, **kwargs)
    self.metric = metric

calculate_D(m)

The metric compression dimension calculation method.

The compression output vector dimension is \(d = d_{\phi}\). For the majority of metric \(\phi\) studied in this project, the output is typically a scalar, i.e., the dimension \(d_{\phi} = 1\).

Parameters:

Name Type Description Default
m int

The dimension of the input space.

 required

Returns:

Type Description
int

The dimension of the metric compression space.
Source code in tinybig/compression/metric_based_compression.py 
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
def calculate_D(self, m: int):
    r"""
        The metric compression dimension calculation method.

        The compression output vector dimension is $d = d_{\phi}$.
        For the majority of metric $\phi$ studied in this project,
        the output is typically a scalar, i.e., the dimension $d_{\phi} = 1$.

        Parameters
        ----------
        m: int
            The dimension of the input space.

        Returns
        -------
        int
            The dimension of the metric compression space.
    """
    return 1

forward(x, device='cpu', *args, **kwargs)

The forward method of the metric compression function.

It performs the metric based data compression of the input data and returns the compression result.

Formally, given a data instance \(\mathbf{x} \in R^m\) and a provided metric \(\phi: {R}^m \to {R}^{d_{\phi}}\), which transforms it into a dense representation of length \(d_{\phi}\), we can represent the metric based compression function as follows:

\[
    \begin{equation}
    \kappa(\mathbf{x}) = \phi(\mathbf{x}) \in {R}^{d},
    \end{equation}
\]

where the compression output vector dimension is \(d = d_{\phi}\).

Parameters:

Name Type Description Default
x Tensor

The input data vector.

 required
device

The device of the input data vector.

 'cpu'

Returns:

Type Description
Tensor

The compression result.
Source code in tinybig/compression/metric_based_compression.py 
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
def forward(self, x: torch.Tensor, device='cpu', *args, **kwargs):
    r"""
        The forward method of the metric compression function.

        It performs the metric based data compression of the input data and returns the compression result.

        Formally, given a data instance $\mathbf{x} \in R^m$ and a provided metric $\phi: {R}^m \to {R}^{d_{\phi}}$,
        which transforms it into a dense representation of length $d_{\phi}$,
        we can represent the metric based compression function as follows:

        $$
            \begin{equation}
            \kappa(\mathbf{x}) = \phi(\mathbf{x}) \in {R}^{d},
            \end{equation}
        $$

        where the compression output vector dimension is $d = d_{\phi}$.

        Parameters
        ----------
        x: torch.Tensor
            The input data vector.
        device: str, default = 'cpu'
            The device of the input data vector.

        Returns
        -------
        torch.Tensor
            The compression result.
    """
    b, m = x.shape
    x = self.pre_process(x=x, device=device)

    assert self.metric is not None
    compression = self.metric(x).unsqueeze(1)

    assert compression.shape == (b, self.calculate_D(m=m))
    return self.post_process(x=compression, device=device)