/jsDddlZddlmZddlZddlmZddlmZdejfdZ d de fdZ Gd d Z Gd d ej ZdS)N) _get_clones) xywh2xyxymaskc|tj|ddkS)zGiven a padding mask (following pytorch convention, 1s for padded values), returns whether the padding is on the right or not. dimr)longtorchsortall)rs r/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/ultralytics/models/sam/sam3/geometry_encoders.pyis_right_paddedr s? IIKK5:diikkr:::1= = B B D DDF return_indexc|j\}}}|j\}}}||dcxkrM|dcxkr3|dcxkr|dksnJ||dcxkr|dksnJ||dksJ||dksJtjt |dtjt |d|d} |d} | | z} ||z} tj| |jd|d| dddfk} tj | ||f|j|j }||d|ddddf<tj||jdddfd|}|| dz}| d|dddddf dd||}|r|| |fS|| fS) ab Concatenates two right-padded sequences, such that the resulting sequence is contiguous and also right-padded. Following pytorch's convention, tensors are sequence first, and the mask are batch first, with 1s for padded values. :param seq1: A tensor of shape (seq1_length, batch_size, hidden_size). :param mask1: A tensor of shape (batch_size, seq1_length). :param seq2: A tensor of shape (seq2_length, batch_size, hidden_size). :param mask2: A tensor of shape (batch_size, seq2_length). :param return_index: If True, also returns the index of the ids of the element of seq2 in the concatenated sequence. This can be used to retrieve the elements of seq2 :return: A tuple (concatenated_sequence, concatenated_mask) if return_index is False, otherwise (concatenated_sequence, concatenated_mask, index). rzMask is not right paddedrr)deviceNrdtype) shapesizer _assertrsumarangerrepeatzerosrscatterexpand)seq1mask1seq2mask2r seq1_length batch_size hidden_size seq2_lengthactual_seq1_lengthsactual_seq2_lengths final_lengths max_lengthconcatenated_maskconcatenated_sequenceindexs rconcat_padded_sequencesr0s",0:(K[+/:(K[ 1 W W W W1 W W W WA W W W W%**UV-- W W W W W W $))A,, 6 6 6 6$))A,, 6 6 6 6 6 6 %**Q-- ' ' ' ' %**Q-- ' ' ' ' M/%((*DEEE M/%((*DEEE!6,,2,..!6,,2,..'*==M{*J Z 444T:AA*aPPTabcbcbceibiTjj "KZ(MVZVaimisttt04,;,111,- LT[ 9 9 9!!!T' B I I!Z X XE '- -E199!U111aaa:=N=U=UVXZ\^i=j=jlpqq?$&7>> "3 33rc"eZdZdZddZddZdS)PromptaDUtility class to manipulate geometric prompts. We expect the sequences in pytorch convention, that is sequence first, batch second The dimensions are expected as follows: box_embeddings shape: N_boxes x B x C_box box_mask shape: B x N_boxes. Can be None if nothing is masked out point_embeddings shape: N_points x B x C_point point_mask shape: B x N_points. Can be None if nothing is masked out mask_embeddings shape: N_masks x B x 1 x H_mask x W_mask mask_mask shape: B x N_masks. Can be None if nothing is masked out We also store positive/negative labels. These tensors are also stored batch-first If they are None, we'll assume positive labels everywhere box_labels: long tensor of shape N_boxes x B point_labels: long tensor of shape N_points x B mask_labels: long tensor of shape N_masks x B Nc|d|_d|_d|_dS|jd}|jd}|j}|"t j|||t j}|"t j|||t j }t|jdd||gksJd|d|d|j|jd d ksJd |jd t|j||gksJd |d|d |jt|j||gksJd|d|d |j|j|ksJd|d|j|j|ksJd|d|j|j|ksJd|d|j||_||_||_dS)zInitialize the Prompt object.Nrrrrz.Wrong dimension for box embeddings. Expected [z, z , *] got rz3Expected box embeddings to have 4 coordinates, got z(Wrong dimension for box mask. Expected [z] got z*Wrong dimension for box labels. Expected [z(Expected box embeddings to be on device , got z"Expected box mask to be on device z$Expected box labels to be on device ) box_embeddings box_labelsbox_maskrrr onesr rboollist)selfr6r8r7 box_seq_lenbsrs r__init__zPrompt.__init__Ys[  !"&D "DO DM F%*1-  !! $&  KF%*UUUJ  {2{6TTTHN(!,--+r1BBBB o[ o oB o oYgYm o oCBB#B'1,,, \.BVWYBZ \ \-,,HN##K'8888 `r ` `[ ` `PXP^ ` `988J$%%+r):::: d d d d dR\Rb d d;:: $... \v \ \^EZ \ \/..&(((*nv*n*n]e]l*n*n((( F***,tSY,t,takar,t,t***,  $rc2|j||_|jd}|jd}|'tj|||jtj}|'tj|||jtj}||_||_ dS|jjd}|jd|ksJd|d|jd|2tj|jd||jtj}|2tj||jdtj|j}t|jddt|jddksJd |jd |jt|j d |j | d |\|_}|j d |_t|j|j ||\|_|_ dS) aAppend box prompts to existing prompts. Args: boxes (torch.Tensor): Tensor of shape (N_new_boxes, B, 4) with normalized box coordinates. labels (torch.Tensor | None): Optional tensor of shape (N_new_boxes, B) with positive/negative labels. mask (torch.Tensor | None): Optional tensor of shape (B, N_new_boxes) for attention mask. NrrrzBatch size mismatch: expected r5rrrzShape mismatch between boxes z and labels r)r6rr r9rr rr:r7r8r;r0 unsqueezesqueeze)r<boxeslabelsrr>r=_s r append_boxeszPrompt.append_boxess   &"'D QB+a.K~KELPUPZ[[[|{2{5Z A5ELYYYDEKO$$V\"1"-=(>(>>>> SEK S SV\ S S?>> 5 O % %b ) )4=&:J:J2:N:NPT  /11"55-DTEXZ^Zginpt-u-u*T]]]r)NNN)NN)__name__ __module__ __qualname____doc__r?rGrrr2r2KsM  ,%,%,%,%\)v)v)v)v)v)vrr2ceZdZdZ ddedededed ed ed ejd ed ededeffd ZdZ de j fdZ dde fdZxZS)SequenceGeometryEncodera]Encoder for geometric box prompts. Assumes boxes are passed in the "normalized CxCyWH" format. Boxes can be encoded with any of the three possibilities: - direct projection: linear projection from coordinate space to d_model - pooling: RoI align features from the backbone - pos encoder: position encoding of the box center These three options are mutually compatible and will be summed if multiple are selected. As an alternative, boxes can be encoded as two corner points (top-left and bottom-right). The encoded sequence can be further processed with a transformer. TFencode_boxes_as_pointsboxes_direct_project boxes_pool boxes_pos_encd_model num_layerslayerroi_sizeadd_clsadd_post_encode_proj use_act_ckptc t||_||_||_| |_|jrdnd} t j| |j|_ d|_ | r*t jd|j|_ |r.tj d|j|_ d|_ d|_n|s|s |s Jdd|_ d|_ d|_d|_d|_d|_|rtj d|j|_|r*tj|j|j|j|_|r'tj |jdz|j|_d|_| rBtj |j|j|_tj|j|_tj|_|j |jtj|j|_d|_|dkr?| s Jdt3|||_tj|j|_| |_dS) z'Initialize the SequenceGeometryEncoder.rNrz,Error: need at least one way to encode boxesr4rzGIt's currently highly recommended to add a CLS when using a transformer)superr?rTpos_encrPrWr nn Embedding label_embed cls_embedLinearpoints_direct_projectpoints_pool_projectpoints_pos_enc_projectrQboxes_pool_projectboxes_pos_enc_projectConv2d final_proj LayerNormnormIdentity img_pre_normencoder encode_normrZ)r<rPrQrRrSrTr^rUrVrWrXrYrZ num_labels __class__s rr?z SequenceGeometryEncoder.__init__sH   &<#  5> e ee e e7%eZ88DK!|DL99D (rc|||j}||}||z|fS)z?Encode points (used when boxes are converted to corner points).)rdtorrar )r<points points_mask points_labels img_feats points_embed type_embeds r_encode_pointsz&SequenceGeometryEncoder._encode_points sV11&))IO2L2LMM %%m&8&8&:&:;; L(+55rrxcd}|jdd\}}|j/|||j}|}|j\|jdd\} } t ||j} t j| | | | g| j} | | jd} | ddd} | | z} tj || d dd |j} t!| j||z|j|j|jgksJ|| }| |||j d d}||}n||z}|j|d \}}}}|j||||}| |jd |jd|jd }|||j}||}n||z}||}||z|fS) z/Encode boxes using configured encoding methods.Nr)rT)r non_blockingrr4rr)rrQrtrrgrr tensorrview torchvisionops roi_align transposeunbindrWr;rTrhr^ encode_boxesflattenrar )r<rD boxes_mask boxes_labelsrx boxes_embedn_boxesr>projHW boxes_xyxyscalesampledcxcywhencrzs r _encode_boxesz%SequenceGeometryEncoder._encode_boxess k"1"o   $ 0,,UXXio-F-FGGDK  " .?233'DAq#588IO#<#<==JL!Q1Z5EFFFEHHJ$5DHIIEJJq!Q''E#e+J"o// :;O;OPQST;U;U;\;\]^;_;_aeanooG &&W    + **733D99R$,77AA!QGGD"" )D0  % 1 <<++LBAq,++BJJLL"**,, UVU^U^U`U`aaC((5;q>5;q>39R=IIC--cffY_.E.EFFD"" )D0 %%l&7&7&9&9:: K'33rN geo_promptc||j}|j}|j}|d}||dntj|} |js|jrt|t|ksJ|d} || } |d\} } | j d| | zksJ| j dd\} }| ddd} | | || | } | }|j r||j ddksJt|}|dd\}}|dz}|dz}tj||gd }tj||gd }tj||gd }||||| \}}n||||| \}}|j d}|j d|ksJ|jr|jj dd|jd|d}tj|d|j|j }t3||||\}}|j(|||}|j.|jD]}||||| }||}||fS)aEncode geometric box prompts. Args: geo_prompt (Prompt): Prompt object containing box embeddings, masks, and labels. img_feats (list[torch.Tensor]): List of image features from backbone. img_sizes (list[tuple[int, int]]): List of (H, W) tuples for each feature level. img_pos_embeds (list[torch.Tensor] | None): Optional position embeddings for image features. Returns: Tuple of (encoded_embeddings, attention_mask) rNrr}rrr4) split_sizer r)rurvrwrx)rDrrrxrA)tgtmemorytgt_key_padding_maskpos)r6r8r7r zeros_likererglenrnrpermuterrPrsplitcatr{rrbweightrTrrrrr0rjrlrorp)r<rrx img_sizesimg_pos_embedsrDrrseq_first_img_featsseq_first_img_pos_embeds cur_img_featrrNCrtop_left bottom_right labels_tl labels_brrurwrv final_embeds final_maskr>clscls_masklays rforwardzSequenceGeometryEncoder.forwardFs;)( !, 'm"0" %y>>S^^3333$R=L,,\::LR=DAq%a(AE1111%bcc*DAq'//1a88L',,Q1a88L$I  & $RA)=)=)=="5))J%/%5%5%5%K%K "Hl%q(I$q(IY,7Q???F!Iy)& %.',,Q4<@@GG2qQQC{2q 0@IZ[[[H'>|ZY\^f'g'g $L* ? &99T__\%B%BCCL ; "{  "s$.)30      ++L99LZ''r)rOTTF)N)rHrIrJrKr:intr_Moduler?r{r Tensorrr2r __classcell__)rrs@rrNrNs2  0%)"G)G) $G)#G) G)  G)  G)G)yG)G)G)#G)G)G)G)G)G)G)R66604 04040404dY(Y(&Y(Y(Y(Y(Y(Y(Y(Y(rrN)F)r torch.nnr_rultralytics.nn.modules.utilsrultralytics.utils.opsrrrr:r0r2rrNrLrrrs  444444++++++E%,EEEE4444D44444444nevevevevevevevevPl(l(l(l(l(bil(l(l(l(l(r