jTddlmZddlZddlmZddlmZddlmZGddejZ Gd d ejZ Gd d e Z d Z dS)) annotationsN)nn) _get_clonesget_valid_ratiocpeZdZdZ d d!fd Z d"d#dZ d$d%dZ d$d%dZxZS)&TransformerEncoderLayeraTransformer encoder layer that performs self-attention followed by cross-attention. This layer was previously called TransformerDecoderLayer but was renamed to better reflect its role in the architecture. It processes input sequences through self-attention and then cross-attention with another input (typically image features). The layer supports both pre-norm and post-norm configurations, as well as positional encoding at different stages of the attention mechanism. Nd_modelintdim_feedforwarddropoutfloatpos_enc_at_attnboolpos_enc_at_cross_attn_keyspos_enc_at_cross_attn_queriespre_normself_attention nn.Modulecross_attentionc &t||_||_||_|pt jddd|_| pt jddd|_t j |||_ t j ||_ t j |||_ t j||_t j||_t j||_t j ||_t j ||_t j ||_t j|_||_||_||_||_d|_dS)aInitialize a transformer encoder layer. Args: d_model: Model dimension/hidden size dim_feedforward: Dimension of the feedforward network dropout: Dropout probability pos_enc_at_attn: Whether to add positional encodings at self-attention pos_enc_at_cross_attn_keys: Whether to add positional encodings to keys in cross-attention pos_enc_at_cross_attn_queries: Whether to add positional encodings to queries in cross-attention pre_norm: Whether to use pre-norm (True) or post-norm (False) architecture self_attention: Self-attention module cross_attention: Cross-attention module for attending to image features g?) num_headsr embed_dimN)super__init__r r dropout_valuerMultiheadAttention self_attncross_attn_imageLinearlinear1Dropoutrlinear2 LayerNormnorm1norm2norm3dropout1dropout2dropout3ReLU activationrrrr layer_idx) selfr r rrrrrrr __class__s h/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/ultralytics/models/sam/sam3/encoder.pyrz TransformerEncoderLayer.__init__sJ2  .$'i2+@1VYeh+i+i+i / q23HST^amp3q3q3qy/:: z'** y':: \'** \'** \'**  7++  7++  7++ '))  .-J**D'tgt torch.Tensormemorytgt_mask memory_masktgt_key_padding_maskmemory_key_padding_maskpos query_posreturnc |jr||zn|x} } || | |||dd} ||| z}||}||jr||zn||jr||zn||||dd} ||| z}||}| | | | |} || | z}||}|S)aKForward pass for post-norm architecture. In post-norm architecture, normalization is applied after attention and feedforward operations. Args: tgt (torch.Tensor): Input tensor to be processed. memory (torch.Tensor): Memory tensor for cross-attention. tgt_mask (torch.Tensor): Mask for self-attention. memory_mask (torch.Tensor): Mask for cross-attention. tgt_key_padding_mask (torch.Tensor): Key padding mask for self-attention. memory_key_padding_mask (torch.Tensor): Key padding mask for cross-attention. pos (torch.Tensor): Positional encoding for memory. query_pos (torch.Tensor): Positional encoding for query. **kwargs (Any): Additional keyword arguments. Returns: Processed tensor F)value attn_maskkey_padding_mask need_weightsr)querykeyr@rArBrC)rr!r+r(r"rrr,r)r&rr/r$r-r*) r1r5r7r8r9r:r;r<r=kwargsqktgt2s r3 forward_postz$TransformerEncoderLayer.forward_postOsV<$(#7@iS@A~~ qxBVej   DMM$'''jjoo$$%)%GP# //S $ ?K V!4 %    DMM$'''jjoo||DLLc9J9J)K)KLLMMDMM$'''jjoo r4Fdacc |rF|jddzdksJ||jddzd} |d|jddz}||} |jr| | zn| x} } || | | ||d} ||| z}|rt j|| fd}||} | | j }| |j r| | zn| |j r||zn||||d} ||| z}||} ||||| } ||| z}|S)aForward pass for pre-norm architecture. In pre-norm architecture, normalization is applied before attention and feedforward operations. Args: tgt: Input tensor to be processed memory: Memory tensor for cross-attention dac: Whether to use Divide-and-Conquer attention tgt_mask: Mask for self-attention memory_mask: Mask for cross-attention tgt_key_padding_mask: Key padding mask for self-attention memory_key_padding_mask: Key padding mask for cross-attention pos: Positional encoding for memory query_pos: Positional encoding for query Returns: Processed tensor rN)r@rArBdim)rDrEr@rArB)shaper( contiguousrr!r+torchcatr)todtyper"rrr,r*r&rr/r$r-)r1r5r7rKr8r9r:r;r<r= other_tgtrIrGrHs r3 forward_prez#TransformerEncoderLayer.forward_pres<  +9Qr6)FNNNNNN)r5r6r7r6rKrr8r6r9r6r:r6r;r6r<r6r=r6r>r6) __name__ __module__ __qualname____doc__rrJrWrZ __classcell__r2s@r3r r s&%)%)3333333r"&$(-104 "&77777z!%$(-104 "&77777z!%$(-104 "&) ) ) ) ) ) ) ) ) r4r cDeZdZdZ ddfd Zd Z dddZxZS) TransformerEncoderaTransformer encoder that processes multi-level features. This encoder takes multi-level features (e.g., from a backbone network) and processes them through a stack of transformer encoder layers. It supports features from multiple levels (e.g., different resolutions) and can apply activation checkpointing for memory efficiency during training. Args: layer: The encoder layer to be stacked multiple times num_layers: Number of encoder layers to stack d_model: Model dimension/hidden size num_feature_levels: Number of feature levels to process frozen: Whether to freeze the parameters of this module use_act_checkpoint: Whether to use activation checkpointing during training Flayerr num_layersr r num_feature_levelsfrozenruse_act_checkpointctt|||_||_||_d|_|dkr,tjtj |||_|r,| D]}| d||_ t|jD] \}}||_ dS)z#Initialize the transformer encoder.NrF)rrrlayersrdre level_embedr ParameterrRTensor parametersrequires_grad_rg enumerater0) r1rcrdr rerfrgpr0r2s r3rzTransformerEncoder.__init__s !%44 $"4  ! !!|EL9KW,U,UVVD   (__&& ( (  ''''"4 !*$+ 6 6 ( ( Iu'EOO ( (r4ct||jks Jdg}g}g}g}|duo |ddu}tt|||D]\} \} } } | j\} } }}||f}||| ddd} |r| d} | ddd} |j&| |j|  dddz}n| }|||| |r||  tj |d}|rtj |dnd}tj |d}tj |tj |j}tj |d|ddddf}|r tjd |Dd}n4tj|jd|jdf|j|j }||||||fS) z5Prepare multi-level features for transformer encoder.z:mismatch between expected and received # of feature levelsNrrMr)rUdevice)rc,g|]}t|Sr).0ms r3 zCTransformerEncoder._prepare_multilevel_features..As 'J'J'Jq(:(:'J'J'Jr4)rsrU)lenreroziprPappendflatten transposerjviewrRrStensorlongrs new_zerosprodcumsumstackonesrU)r1srcsmasks pos_embeds src_flatten mask_flattenlvl_pos_embed_flattenspatial_shapeshas_masklvlsrcmask pos_embed_hw spatial_shape lvl_pos_embedlevel_start_index valid_ratioss r3_prepare_multilevel_featuresz/TransformerEncoder._prepare_multilevel_featuress4yyD33335q333  "$=q)=+4Suj5Q5Q+R+R * * 'C'#tYJAq!QFM  ! !- 0 0 0++a..**1a00C '||A!))!,,66q!<B")!'L   !      r4Nrlist[torch.Tensor]src_key_padding_maskslist[torch.Tensor] | Noner<promptr6prompt_key_padding_maskencoder_extra_kwargs dict | Noner>Ytuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]clt||jks Jd|t||jksJ|t||jksJ||||\}}} } } } |} |jD]k}i}t |t sJ||d<||d<| |d<| |d<||d<|jr|js Jd||||d i|} l| d d || d d nd| d d | | | fS) aProcess multi-level features through the transformer encoder. Args: src: List of multi-level features, each with shape (batch_size, channels, height, width) src_key_padding_masks: List of padding masks for each feature level, each with shape (batch_size, height, width) pos: List of positional embeddings for each feature level, each with shape (batch_size, channels, height, width) prompt: Optional text/prompt features to attend to, with shape (seq_len, batch_size, d_model) prompt_key_padding_mask: Optional padding mask for prompt, with shape (batch_size, seq_len) encoder_extra_kwargs: Optional additional arguments to pass to each encoder layer Returns: A tuple containing: - output: Processed features with shape (seq_len, batch_size, d_model) - key_padding_masks_flatten: Flattened padding masks - lvl_pos_embed_flatten: Flattened positional embeddings - level_start_index: Starting indices for each feature level - spatial_shapes: Spatial dimensions of each feature level - valid_ratios: Valid ratios for each feature level z#must be equal to num_feature_levelsNr7r;r=r5r:z&activation ckpt not enabled in encoderrrru) ryrerri isinstancer trainingrgupdater})r1rrr<rrrrkey_padding_masks_flattenrrrroutputrc layer_kwargss r3rZzTransformerEncoder.forwardRs<3xx422224Y222 ,,--1HHHHH ?s88t66666  - -c3H# N N  % !   [ + +ELe%<== = ==%+L "6ML2 3(=L %"(L 3LL/ 0} Y.XX0XXX.#/##$8999U**\**FF   Q " ":S:_ & 0 0A 6 6 6ei ! + +Aq 1 1      r4)FF) rcrrdr r r rer rfrrgrNNNNN)rrrrr<rrr6rr6rrr>r)r[r\r]r^rrrZr_r`s@r3rbrbs  *#((((((((>4 4 4 r<@)-#04,0E E E E E E E E E r4rbcDeZdZdZ ddfd Z d d!fd ZxZS)"TransformerEncoderFusionaTransformer encoder that fuses text and image features. This encoder extends TransformerEncoder to handle both text and image features, with the ability to add pooled text features to image features for better cross-modal fusion. It supports torch.compile for performance optimization. Args: layer (nn.Module): The encoder layer to be stacked multiple times. num_layers (int): Number of encoder layers to stack. d_model (int): Model dimension/hidden size. num_feature_levels (int): Number of feature levels to process. add_pooled_text_to_img_feat (bool): Whether to add pooled text features to image features. pool_text_with_mask (bool): Whether to use the mask when pooling text features. compile_mode (str | None): Mode for torch.compile, or None to disable compilation. **kwargs (Any): Additional arguments to pass to the parent class. TFNrcrrdr r readd_pooled_text_to_img_featrpool_text_with_mask compile_mode str | Nonec tj||||fi|||_|jrtj|||_||_|#tj|j |d|_ dSdS)z:Initialize the transformer encoder with text-image fusion.NT)mode fullgraph) rrrrr#text_pooling_projrrRcompilerZ) r1rcrdr rerrrrFr2s r3rz!TransformerEncoderFusion.__init__s            ,G(  + A%'Yw%@%@D "#6  # =LTXYYYDLLL $ #r4rrrr6src_key_padding_maskrsrc_posr feat_sizeslist[int] | Nonerrc |djd}|t|t|ksJ|dgt|z}t|D]\} \} } || | | |ddddd|| <|| | | |ddddd|| <|| 2|| | | |dddnd|| <n#t d|Ds Jd|jr?t|||j| d fd |D}t |||| dd|| \} } }}}}| | |||||d S) z@Forward pass for the transformer encoder with text-image fusion.rrNrrrMc3,K|]}|jdkVdS)NrN)rvxs r3 z3TransformerEncoderFusion.forward..s(//aquz//////r4z&expected list of (bs, c, h, w) tensors).NNc:g|]}|Sru)add_)rvr pooled_texts r3rxz4TransformerEncoderFusion.forward..s%4441166+&&444r4)rr<rrr)r7 padding_maskr memory_textrrr) rPryroreshapepermuteallrpool_text_featrrrrZr})r1rrrrrrrbsirroutrrrrrrr2s @r3rZz TransformerEncoderFusion.forwards%V\!_  !z??c#hh....#+(,vC'8$&z22   6AqQ1b"55==aAqIIA$QZ//1b"==EEaAqQQ ,A.:)+33Aq"==EEaANNN%Q'' //3///// Y Y1Y Y Y/  + 5(1H$JbccK00==oNK4444444C GGOO "6##Aq))$;!5     % !   5.!!2,(   r4)TFN)rcrrdr r r rer rrrrrrr)rrrr6rrrrrr6rrrr)r[r\r]r^rrZr_r`s@r3rrs,-1$)#'ZZZZZZZ<;?-104'+,0: : : : : : : : : : : r4rcT|s|dS|dksJ|ddd}t jt j|dd}||zd|z }|S)z;Mean-pool the prompt embeddings over the valid tokens only.rrNrMr).Ng?)min)meanrOrrrRclampsum)r prompt_maskpool_with_maskis_valid num_validrs r3rrs "{{q{!!! ??   ! ! ! ! ##%%--a33I>H EIhA666C@@@IH$))a)009rs #""""" 444444''''''[ [ [ [ [ bi[ [ [ |j j j j j j j j Ze e e e e 1e e e Pr4