mean_fusion

Bases: metric_fusion

A fusion mechanism that computes the element-wise mean across input tensors.

Notes

Formally, given the input interdependence matrices \(\mathbf{A}_1, \mathbf{A}_2, \ldots, \mathbf{A}_k \in R^{m \times n}\) of identical shapes, we can represent their fusion output as

\[
    \begin{equation}
    \text{fusion}(\mathbf{A}_1, \mathbf{A}_2, \cdots, \mathbf{A}_k) = \mathbf{A}  \in R^{m \times n},
    \end{equation}
\]

where the entry \(\mathbf{A}(i, j)\) (for \(i \in \{1, 2, \cdots, m\}\) and \(j \in \{1, 2, \cdots, n\}\)) can be represented as

\[
    \begin{equation}
    \mathbf{A}(i, j) = mean \left( \mathbf{A}_1(i,j), \mathbf{A}_2(i,j), \cdots, \mathbf{A}_k(i,j)  \right).
    \end{equation}
\]

Methods:

Name	Description
__init__	Initializes the mean fusion function.

Source code in tinybig/fusion/metric_fusion.py

class mean_fusion(metric_fusion):
    r"""
        A fusion mechanism that computes the element-wise mean across input tensors.

        Notes
        ----------

        Formally, given the input interdependence matrices $\mathbf{A}_1, \mathbf{A}_2, \ldots, \mathbf{A}_k \in R^{m \times n}$ of identical shapes,
        we can represent their fusion output as

        $$
            \begin{equation}
            \text{fusion}(\mathbf{A}_1, \mathbf{A}_2, \cdots, \mathbf{A}_k) = \mathbf{A}  \in R^{m \times n},
            \end{equation}
        $$

        where the entry $\mathbf{A}(i, j)$ (for $i \in \{1, 2, \cdots, m\}$ and $j \in \{1, 2, \cdots, n\}$) can be represented as

        $$
            \begin{equation}
            \mathbf{A}(i, j) = mean \left( \mathbf{A}_1(i,j), \mathbf{A}_2(i,j), \cdots, \mathbf{A}_k(i,j)  \right).
            \end{equation}
        $$

        Methods
        -------
        __init__(...)
            Initializes the mean fusion function.
    """
    def __init__(self, name: str = "mean_fusion", *args, **kwargs):
        """
            Initializes the mean fusion function.

            Parameters
            ----------
            name : str, optional
                Name of the fusion function. Defaults to "mean_fusion".
            *args : tuple
                Additional positional arguments for the parent class.
            **kwargs : dict
                Additional keyword arguments for the parent class.
        """
        super().__init__(name=name, metric=batch_mean, *args, **kwargs)

__init__(name='mean_fusion', *args, **kwargs)

Initializes the mean fusion function.

Parameters:

Name	Type	Description	Default
name	str	Name of the fusion function. Defaults to "mean_fusion".	'mean_fusion'
*args	tuple	Additional positional arguments for the parent class.	()
**kwargs	dict	Additional keyword arguments for the parent class.	{}

Source code in tinybig/fusion/metric_fusion.py

def __init__(self, name: str = "mean_fusion", *args, **kwargs):
    """
        Initializes the mean fusion function.

        Parameters
        ----------
        name : str, optional
            Name of the fusion function. Defaults to "mean_fusion".
        *args : tuple
            Additional positional arguments for the parent class.
        **kwargs : dict
            Additional keyword arguments for the parent class.
    """
    super().__init__(name=name, metric=batch_mean, *args, **kwargs)