average_fusion

Bases: weighted_summation_fusion

A fusion mechanism that combines inputs using simple averaging.

It inherits from the weighted summation fusion class, and the \(k\) inputs are treated with equal importance with weight value \(1/k\).

Attributes:

Name	Type	Description
weights	Tensor	Predefined weights, initialized to 1/N for all inputs, where N is the number of inputs.

Methods:

Name	Description
__init__	Initializes the average fusion function.

Source code in tinybig/fusion/basic_fusion.py

class average_fusion(weighted_summation_fusion):
    """
        A fusion mechanism that combines inputs using simple averaging.

        It inherits from the weighted summation fusion class, and the $k$ inputs are treated with equal importance with weight value $1/k$.

        Attributes
        ----------
        weights : torch.Tensor
            Predefined weights, initialized to 1/N for all inputs, where N is the number of inputs.

        Methods
        -------
        __init__(...)
            Initializes the average fusion function.
    """
    def __init__(self, dims: list[int] | tuple[int], name: str = "average_fusion", require_parameters: bool = False, *args, **kwargs):
        """
            Initializes the average fusion function.

            Parameters
            ----------
            dims : list[int] | tuple[int]
                Dimensions of the input tensors.
            name : str, optional
                Name of the fusion function. Defaults to "average_fusion".
            require_parameters : bool, optional
                Whether parameters are required. Defaults to False.
            *args : tuple
                Additional positional arguments for the parent class.
            **kwargs : dict
                Additional keyword arguments for the parent class.
        """
        super().__init__(dims=dims, weights=1.0/len(dims)*torch.ones(len(dims)), name=name, require_parameters=False, *args, **kwargs)

__init__(dims, name='average_fusion', require_parameters=False, *args, **kwargs)

Initializes the average fusion function.

Parameters:

Name	Type	Description	Default
dims	list[int] | tuple[int]	Dimensions of the input tensors.	required
name	str	Name of the fusion function. Defaults to "average_fusion".	'average_fusion'
require_parameters	bool	Whether parameters are required. Defaults to False.	False
*args	tuple	Additional positional arguments for the parent class.	()
**kwargs	dict	Additional keyword arguments for the parent class.	{}

Source code in tinybig/fusion/basic_fusion.py

def __init__(self, dims: list[int] | tuple[int], name: str = "average_fusion", require_parameters: bool = False, *args, **kwargs):
    """
        Initializes the average fusion function.

        Parameters
        ----------
        dims : list[int] | tuple[int]
            Dimensions of the input tensors.
        name : str, optional
            Name of the fusion function. Defaults to "average_fusion".
        require_parameters : bool, optional
            Whether parameters are required. Defaults to False.
        *args : tuple
            Additional positional arguments for the parent class.
        **kwargs : dict
            Additional keyword arguments for the parent class.
    """
    super().__init__(dims=dims, weights=1.0/len(dims)*torch.ones(len(dims)), name=name, require_parameters=False, *args, **kwargs)