elu2"`
        dropout_prob (`float`, *optional*, defaults to 0.1):
            The dropout probability for EMA self-attention
        hidden_dropout_prob (`float`, *optional*, defaults to 0.1):
            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
        attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1):
            The dropout ratio for the attention probabilities.
        use_feature_dropout (`bool`, *optional*, defaults to `False`):
            Whether to use feature-based (`True`) or standard dropout (`False`)
        use_normalized_ffn (`bool`, *optional*, defaults to `True`):
            Whether to use the normalized feed-forward sub-layer in Mega blocks (`True`) or pass Mega encoder output
            as-is (`False`)
        nffn_hidden_size (`int`, *optional*, defaults to 256):
            If using the normalized feed-forward network (NFFN) layer within Mega (`use_normalized_ffn = True`), this
            is the hidden size of the NFFN
        normalize_before_ffn (`bool`, *optional*, defaults to `True`):
            Whether to normalize before (`True`) or after (`False`) the feed-forward portion of NFFN
        nffn_activation_dropout_prob (`float`, *optional*, defaults to 0.1):
            The dropout ratio for the NFFN component.
        max_positions (`int`, *optional*, defaults to 2048):
            The maximum sequence length to use for positional representations. For `"simple"` relative positional bias,
            this is a hard limit on input length; `"rotary"` relative positional bias will extrapolate to longer
            sequences
        add_token_type_embeddings (`bool`, *optional*, defaults to `True`):
            Whether to account for token types in embeddings. Left as optional to maintain compatibility with original
            implementation while adding support for token types.
        type_vocab_size (`int`, *optional*, defaults to 2):
            The vocabulary size of the `token_type_ids` passed when calling [`MegaModel`]. Only used if
            `add_token_type_embeddings = True`
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        ema_delta_alpha_range (`float`, *optional*, defaults to 0.2):
            The standard deviation for initializing the delta (damping factor) and alpha (decay factor) parameters in
            MegaMultiDimensionDampedEma.
        ema_beta_range (`float`, *optional*, defaults to 0.02):
            The standard deviation for initializing the beta parameter (expansion matrix) in
            MegaMultiDimensionDampedEma.
        ema_gamma_omega_range (`float`, *optional*, defaults to 1.0):
            The standard deviation for initializing the gamma (projection matrix) and omega (residual weight)
            parameters in MultiDimensionEMA.
        relative_positional_bias (`str`, *optional*, defaults to `"rotary"`):
            Type of relative positional encoding. Choose one of `"rotary"` or `"simple"`. If `"simple"` is selected,
            `max_positions` is used as a limit on input size, while `"rotary"` extrapolates beyond `max_positions`.
        is_decoder (`bool`, *optional*, defaults to `False`):
            Whether the model is used as a decoder or not. If `False`, the model is used as an encoder.
        use_cache (`bool`, *optional*, defaults to `True`):
            Whether or not the model should return the last key/values attentions (not used by all models). Only
            relevant if `config.is_decoder=True`.
        classifier_dropout (`float`, *optional*):
            The dropout ratio for the classification head.
        add_lm_hidden_dense_layer (`bool`, *optional*, defaults to `True`):
            Whether to include a hidden layer for projection between encoder outputs and LM heads (`True`) or pass
            hidden states directly to LM head (`False`). Remains optional for compatibility with original
            implementation

    Examples:

    ```python
    >>> from transformers import MegaConfig, MegaModel

    >>> # Initializing a Mega configuration
    >>> configuration = MegaConfig()

    >>> # Initializing a model (with random weights) from the configuration
    >>> model = MegaModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config
    ```Ú