j$:ddlmZddlZddlZddlmZmZddlmZGddejZ GddejZ Gd d ejZ dS) ) annotationsN)Tensornn)MLPBlockc<eZdZdZejdfdfd ZddZxZS)TwoWayTransformeraA Two-Way Transformer module for simultaneous attention to image and query points. This class implements a specialized transformer decoder that attends to an input image using queries with supplied positional embeddings. It's useful for tasks like object detection, image segmentation, and point cloud processing. Attributes: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for input embeddings. num_heads (int): Number of heads for multihead attention. mlp_dim (int): Internal channel dimension for the MLP block. layers (nn.ModuleList): List of TwoWayAttentionBlock layers composing the transformer. final_attn_token_to_image (Attention): Final attention layer from queries to image. norm_final_attn (nn.LayerNorm): Layer normalization applied to final queries. Methods: forward: Process image and point embeddings through the transformer. Examples: >>> transformer = TwoWayTransformer(depth=6, embedding_dim=256, num_heads=8, mlp_dim=2048) >>> image_embedding = torch.randn(1, 256, 32, 32) >>> image_pe = torch.randn(1, 256, 32, 32) >>> point_embedding = torch.randn(1, 100, 256) >>> output_queries, output_image = transformer(image_embedding, image_pe, point_embedding) >>> print(output_queries.shape, output_image.shape) depthint embedding_dim num_headsmlp_dim activationtype[nn.Module]attention_downsample_ratereturnNonec t||_||_||_||_t j|_t|D]3}|j t||||||dk4t||||_ t j||_dS)abInitialize a Two-Way Transformer for simultaneous attention to image and query points. Args: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for input embeddings. num_heads (int): Number of heads for multihead attention. Must divide embedding_dim. mlp_dim (int): Internal channel dimension for the MLP block. activation (type[nn.Module], optional): Activation function to use in the MLP block. attention_downsample_rate (int, optional): Downsampling rate for attention mechanism. r)r r rrrskip_first_layer_pedownsample_rateN)super__init__r r r rr ModuleListlayersrangeappendTwoWayAttentionBlock Attentionfinal_attn_token_to_image LayerNormnorm_final_attn) selfr r r rrri __class__s o/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/ultralytics/models/sam/modules/transformer.pyrzTwoWayTransformer.__init__(s&  *" moo u  A K  $"/'#).G)*a     *3=)]v)w)w)w&!|M::image_embedding torch.Tensorimage_pepoint_embedding!tuple[torch.Tensor, torch.Tensor]cj|dddd}|dddd}|}|}|jD]}|||||\}}||z}||z}||||} || z}||}||fS)aProcess image and point embeddings through the Two-Way Transformer. Args: image_embedding (torch.Tensor): Image to attend to, with shape (B, embedding_dim, H, W). image_pe (torch.Tensor): Positional encoding to add to the image, with same shape as image_embedding. point_embedding (torch.Tensor): Embedding to add to query points, with shape (B, N_points, embedding_dim). Returns: queries (torch.Tensor): Processed point embeddings with shape (B, N_points, embedding_dim). keys (torch.Tensor): Processed image embeddings with shape (B, H*W, embedding_dim). r r)querieskeysquery_pekey_peqkv)flattenpermuterr r") r#r(r*r+r/r0layerr4r5attn_outs r&forwardzTwoWayTransformer.forwardQs$*11!44<>> embedding_dim, num_heads = 256, 8 >>> block = TwoWayAttentionBlock(embedding_dim, num_heads) >>> queries = torch.randn(1, 100, embedding_dim) >>> keys = torch.randn(1, 1000, embedding_dim) >>> query_pe = torch.randn(1, 100, embedding_dim) >>> key_pe = torch.randn(1, 1000, embedding_dim) >>> processed_queries, processed_keys = block(queries, keys, query_pe, key_pe) ir Fr r r rrrrrboolrrctt|||_t j||_t||||_t j||_t||||_ t j||_ t j||_ t||||_ ||_dS)aInitialize a TwoWayAttentionBlock for simultaneous attention to image and query points. This block implements a specialized transformer layer with four main components: self-attention on sparse inputs, cross-attention of sparse inputs to dense inputs, MLP block on sparse inputs, and cross-attention of dense inputs to sparse inputs. Args: embedding_dim (int): Channel dimension of the embeddings. num_heads (int): Number of attention heads in the attention layers. mlp_dim (int, optional): Hidden dimension of the MLP block. activation (type[nn.Module], optional): Activation function for the MLP block. attention_downsample_rate (int, optional): Downsampling rate for the attention mechanism. skip_first_layer_pe (bool, optional): Whether to skip positional encoding in the first layer. rN)rrr self_attnrr!norm1cross_attn_token_to_imagenorm2rmlpnorm3norm4cross_attn_image_to_tokenr)r#r r rrrrr%s r&rzTwoWayAttentionBlock.__init__s. "=)<<\-00 )2=)]v)w)w)w&\-00 M7J??\-00 \-00 )2=)]v)w)w)w   r'r/r)r0r1r2r,c|jr||||}n"||z}||||}||z}||}||z}||z}||||}||z}||}||}||z}||}||z}||z}||||}||z}||}||fS)aApply two-way attention to process query and key embeddings in a transformer block. Args: queries (torch.Tensor): Query embeddings with shape (B, N_queries, embedding_dim). keys (torch.Tensor): Key embeddings with shape (B, N_keys, embedding_dim). query_pe (torch.Tensor): Positional encodings for queries with same shape as queries. key_pe (torch.Tensor): Positional encodings for keys with same shape as keys. Returns: queries (torch.Tensor): Processed query embeddings with shape (B, N_queries, embedding_dim). keys (torch.Tensor): Processed key embeddings with shape (B, N_keys, embedding_dim). r3) rrGrHrIrJrKrLrNrM) r#r/r0r1r2r4r:r5mlp_outs r&r;zTwoWayAttentionBlock.forwards/  # )nnw'WnEEGG("A~~Q'~::H(G**W%% h  6M11Ad1CCH$**W%%((7##G#**W%% h  6M11Ag1FFhzz$}r')r r r r rr rrrr rrErr) r/r)r0r)r1r)r2r)rr,r<rCs@r&rr}srD&(g)*$)$7$7$7$7$7$7$7L++++++++r'rcbeZdZdZ ddfd ZeddZeddZddZxZ S)raAn attention layer with downscaling capability for embedding size after projection. This class implements a multi-head attention mechanism with the option to downsample the internal dimension of queries, keys, and values. Attributes: embedding_dim (int): Dimensionality of input embeddings. kv_in_dim (int): Dimensionality of key and value inputs. internal_dim (int): Internal dimension after downsampling. num_heads (int): Number of attention heads. q_proj (nn.Linear): Linear projection for queries. k_proj (nn.Linear): Linear projection for keys. v_proj (nn.Linear): Linear projection for values. out_proj (nn.Linear): Linear projection for output. Methods: _separate_heads: Separate input tensor into attention heads. _recombine_heads: Recombine separated attention heads. forward: Compute attention output for given query, key, and value tensors. Examples: >>> attn = Attention(embedding_dim=256, num_heads=8, downsample_rate=2) >>> q = torch.randn(1, 100, 256) >>> k = v = torch.randn(1, 50, 256) >>> output = attn(q, k, v) >>> print(output.shape) torch.Size([1, 100, 256]) r.Nr r r r kv_in_dim int | Nonerrct||_||n||_||z|_||_|j|zdks Jdt j||j|_t j|j|j|_ t j|j|j|_ t j|j||_ dS)a6Initialize the Attention module with specified dimensions and settings. Args: embedding_dim (int): Dimensionality of input embeddings. num_heads (int): Number of attention heads. downsample_rate (int, optional): Factor by which internal dimensions are downsampled. kv_in_dim (int | None, optional): Dimensionality of key and value inputs. If None, uses embedding_dim. Raises: AssertionError: If num_heads does not evenly divide the internal dim (embedding_dim / downsample_rate). Nrz$num_heads must divide embedding_dim.) rrr rR internal_dimr rLinearq_projk_projv_projout_proj)r#r r rrRr%s r&rzAttention.__init__s$ *&/&;)_<" 9,1113Y111i t/@AA i0ABB i0ABB  $"3]CC r'xr)cz|j\}}}||||||z}|ddS)zGSeparate the input tensor into the specified number of attention heads.r.r )shapereshape transpose)r[r bncs r&_separate_headszAttention._separate_heads,s@'1a IIaIqI~ 6 6{{1a   r'rcz|j\}}}}|dd}|||||zS)z9Recombine separated attention heads into a single tensor.r.r )r]r_r^)r[r`n_headsn_tokens c_per_heads r&_recombine_headszAttention._recombine_heads3sB,-7(7Hj KK1  yyHg &:;;;r'r4r5r6c(||}||}||}|||j}|||j}|||j}|j\}}}}||ddddz}|tj|z }tj |d}||z}| |}| |S)aApply multi-head attention to query, key, and value tensors with optional downsampling. Args: q (torch.Tensor): Query tensor with shape (B, N_q, embedding_dim). k (torch.Tensor): Key tensor with shape (B, N_k, kv_in_dim). v (torch.Tensor): Value tensor with shape (B, N_k, kv_in_dim). Returns: (torch.Tensor): Output tensor after attention with shape (B, N_q, embedding_dim). rr.r )dim) rWrXrYrcr r]r8mathsqrttorchsoftmaxrhrZ)r#r4r5r6_rgattnouts r&r;zAttention.forward:s KKNN KKNN KKNN  DN 3 3  DN 3 3  DN 3 3 g1a199Q1a(((di +++}Tr***Qh##C((}}S!!!r')r.N) r r r r rr rRrSrr)r[r)r r rr))r[rrr)r4r)r5r)r6r)rr)) r=r>r?r@r staticmethodrcrhr;rBrCs@r&rrsB ! $ DDDDDDD<!!!\! <<<\< """"""""r'r) __future__rrmrorrultralytics.nn.modulesrModulerrrr'r&rys#""""" ++++++mmmmm mmm`ppppp29pppfh"h"h"h"h" h"h"h"h"h"r'