jaddlmZddlZddlmZddlmZmZGddejZGddejZ dS) ) annotationsN)nn)MLP LayerNorm2dcHeZdZdZdejddfdfd ZddZddZxZ S) MaskDecoderaDecoder module for generating masks and their associated quality scores using a transformer architecture. This class predicts masks given image and prompt embeddings, utilizing a transformer to process the inputs and generate mask predictions along with their quality scores. Attributes: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. iou_token (nn.Embedding): Embedding for the IoU token. num_mask_tokens (int): Number of mask tokens. mask_tokens (nn.Embedding): Embedding for the mask tokens. output_upscaling (nn.Sequential): Neural network sequence for upscaling the output. output_hypernetworks_mlps (nn.ModuleList): Hypernetwork MLPs for generating masks. iou_prediction_head (nn.Module): MLP for predicting mask quality. Methods: forward: Predict masks given image and prompt embeddings. predict_masks: Internal method for mask prediction. Examples: >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> masks, iou_pred = decoder( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, multimask_output=True ... ) >>> print(f"Predicted masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") transformer_dimint transformer nn.Modulenum_multimask_outputs activationtype[nn.Module]iou_head_depthiou_head_hidden_dimreturnNonec t|_||_||_t jd|_|dz|_t j|j|_ t j t j dzddtdz|t j dzdzdd||_ t jfdt|jD|_t#||j||_dS)aInitialize the MaskDecoder module for generating masks and their associated quality scores. Args: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. activation (type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality.  kernel_sizestridec:g|]}tdzdSrr r.0_r s l/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/ultralytics/models/sam/modules/decoders.py z(MaskDecoder.__init__..M- q q qPQS/?a3G K K q q qN)super__init__r r rr Embedding iou_tokennum_mask_tokens mask_tokens SequentialConvTranspose2droutput_upscaling ModuleListrangeoutput_hypernetworks_mlpsriou_prediction_head)selfr r rrrr __class__s ` r$r)zMaskDecoder.__init__(s?& .&%:"a994q8<(>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> image_emb = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_emb = torch.rand(1, 2, 256) >>> dense_emb = torch.rand(1, 256, 64, 64) >>> masks, iou_pred = decoder(image_emb, image_pe, sparse_emb, dense_emb, multimask_output=True) >>> print(f"Masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") )r7r9r:r;rNr) predict_masksslice) r5r7r9r:r;r<masksiou_pred mask_slices r$forwardzMaskDecoder.forwardRs<,,-%=$; -  x(8HU1d^^^U1a[[ aaaQQQ)*AAAzM*hr'ctjjjjjgd}|d|jddd}tj||fd}tj||jdd}||z}tj||jdd}|j\} } } } |||\} }| dddddf}| ddddj zddf| dd | | | | } |}fdtj D}tj|d}|j\} } } } || | | | | zz | d| | }|}||fS)z`Predict masks and quality scores using image and prompt embeddings via transformer architecture.rdimrNrc Vg|]%}j|dd|ddf&SNr3r"imask_tokens_outr5s r$r%z-MaskDecoder.predict_masks..L- - - LM -D *1 -oaaaAAAg.F G G- - - r')torchcatr+weightr- unsqueezeexpandshaperepeat_interleaver r, transposeviewr0r2stackr4)r5r7r9r:r; output_tokenstokenssrcpos_srcbchwhs iou_token_outupscaled_embedding hyper_in_listhyper_inrBrCrOs` @r$r@zMaskDecoder.predict_masks~s 4>#8$:J:Q"RXYZZZ %//2299:R:XYZ:[]_acdd M+CD!LLL%&6 QQOOO++)(FLOKKKY 1a""388C111a7 QQQQ)=%= >ABmmAq!!&&q!Q22!22377- - - - - QVW[WkQlQl- - - ;}!444'- 1a.33Aq!a%@@@FFq"aQRSS++M::hr')r r r rrr rrrr rr rr) r7r8r9r8r:r8r;r8r<r=rr>) r7r8r9r8r:r8r;r8rr>) __name__ __module__ __qualname____doc__rGELUr)rEr@ __classcell__r6s@r$rr s@&'&(g#&(s(s(s(s(s(s(sT****X%%%%%%%%r'rc leZdZdZdejddddddddddf d'fd Z d(d)d#Z d(d*d$Zd%Z d&Z xZ S)+SAM2MaskDecodera~ Transformer-based decoder for predicting instance segmentation masks from image and prompt embeddings. This class extends the functionality of the MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Attributes: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. iou_token (nn.Embedding): Embedding for IOU token. num_mask_tokens (int): Total number of mask tokens. mask_tokens (nn.Embedding): Embedding for mask tokens. pred_obj_scores (bool): Whether to predict object scores. obj_score_token (nn.Embedding): Embedding for object score token. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. output_upscaling (nn.Sequential): Upscaling layers for output. use_high_res_features (bool): Whether to use high-resolution features. conv_s0 (nn.Conv2d): Convolutional layer for high-resolution features (s0). conv_s1 (nn.Conv2d): Convolutional layer for high-resolution features (s1). output_hypernetworks_mlps (nn.ModuleList): List of MLPs for output hypernetworks. iou_prediction_head (MLP): MLP for IOU prediction. pred_obj_score_head (nn.Linear | MLP): Linear layer or MLP for object score prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. Methods: forward: Predict masks given image and prompt embeddings. predict_masks: Predict instance segmentation masks from image and prompt embeddings. _get_stability_scores: Compute mask stability scores based on IoU between thresholds. _dynamic_multimask_via_stability: Dynamically select the most stable mask output. Examples: >>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) r r Fg?g\(\?r r r rrrrrruse_high_res_featuresr=pred_obj_scorespred_obj_scores_mlpuse_multimask_token_for_obj_ptrrrc t|_||_||_t jd|_|dz|_t j|j|_ | |_ |j rt jd|_ ||_ t j t jdzddtdz|t jdzdzdd||_||_|r@t jdzdd|_t jdzdd|_t jfdt-|jD|_t1||j|||_|j r3t jd|_| rt1dd|_| |_| |_| |_d S) aInitialize the SAM2MaskDecoder module for predicting instance segmentation masks. This decoder extends the functionality of MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Args: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. activation (type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. use_high_res_features (bool): Whether to use high-resolution features. iou_prediction_use_sigmoid (bool): Whether to use sigmoid for IOU prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. pred_obj_scores (bool): Whether to predict object scores. pred_obj_scores_mlp (bool): Whether to use MLP for object score prediction. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. rrrrrc:g|]}tdzdSrr r!s r$r%z,SAM2MaskDecoder.__init__..r&r')sigmoidr N)r(r)r r rrr*r+r,r-rrobj_score_tokenrtr.r/rr0rqConv2dconv_s0conv_s1r1r2r3rr4Linearpred_obj_score_headdynamic_multimask_via_stability!dynamic_multimask_stability_delta"dynamic_multimask_stability_thresh)r5r r rrrrrqiou_prediction_use_sigmoidr~rrrrrsrtr6s ` r$r)zSAM2MaskDecoder.__init__s/L .&%:"a994q8<(>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) )r7r9r:r;rrNrr)r@r~training _dynamic_multimask_via_stabilityrt) r5r7r9r:r;r<rrrBrCrOobject_score_logitssam_tokens_outs r$rEzSAM2MaskDecoder.forward)s=JAE@R@R-%=$;%/ ASA A =x*=  (!!!QRRAAA+&E122HH  1 ($- ("CCE8TTOE88!!!QqS!!!QQQ,'E1Q3'H  5 D 5,QQQU3NN-QQQ!V4Nh0CCCr'c*d}jr:tjjjjjjjgd}d}n,tjjjjjgd}|d|j ddd}tj||fd} |r#tj || j dd} n |j d| j dksJ|} | |z} |j ddks Jdtj || j dd} | j \} } }} | | | \}} |dd|ddf}|dd|dz|dzj zddf| dd| | ||} jr|| }nIj\}}}}}|\}}|||| |z}||||z}fdt#j D}tj|d}|j \} } }}||| | ||zz| d||}|}jr+|dksJ|dddddf}n$d ||j ddz}|||fS) zYPredict instance segmentation masks from image and prompt embeddings using a transformer.rrGrrIz@image_pe should have size 1 in batch dim (from `get_dense_pe()`)Nrc Vg|]%}j|dd|ddf&SrKrLrMs r$r%z1SAM2MaskDecoder.predict_masks..rPr'g$@)rrrQrRrxrSr+r-rTrUrVrWr r,rXrYrqr0r2rZr4r}new_ones) r5r7r9r:r;rrsr[r\r]r^r_r`rarbrcrdredc1ln1act1dc2act2feat_s0feat_s1rfrgrBrCrrOs ` @r$r@zSAM2MaskDecoder.predict_masksms    _!I(/N)$+  MAA!It~'N>U&V\]^^^M%//2299:R:XYZ:[]_acdd M+CD!LLL  #)*:FLOQRSSSCC#)!, Q????"C++~a A%%%'i%%%)(FLOKKKY 1a""388C111a7 QQQAQ1E)E FIJmmAq!!&&q!Q22) I->-F!%!6!6s!;!;  (,(= %CdC0 GW!%cc##c((W*<&=&=!>!> !%cc*<&=&=&G!H!H - - - - - QVW[WkQlQl- - - ;}!444'- 1a.33Aq!a%@@@FFq"aQRSS++M::   Q6666"&":":2aaaAAAg;"G"G  #'):):8>!;La)P)P"P h1DDDr'c,|d}tj||jkd}tj||j kd}tj|dk||z dS)zNCompute mask stability scores based on IoU between upper and lower thresholds.rIrGrg?)flattenrQsumrfloatwhere)r5 mask_logitsarea_iarea_us r$_get_stability_scoresz%SAM2MaskDecoder._get_stability_scoress!))"-- ;)OOUWXXX^^``;$*P)PPVXYYY__aa{6A:v<<>> decoder = SAM2MaskDecoder(...) >>> all_mask_logits = torch.rand(2, 4, 256, 256) # 2 images, 4 masks each >>> all_iou_scores = torch.rand(2, 4) >>> mask_logits, iou_scores = decoder._dynamic_multimask_via_stability(all_mask_logits, all_iou_scores) >>> print(mask_logits.shape, iou_scores.shape) torch.Size([2, 1, 256, 256]) torch.Size([2, 1]) NrrIrGr)device).NN) rQargmaxarangerVrrTrrr expand_as)r5all_mask_logitsall_iou_scoresmultimask_logitsmultimask_iou_scoresbest_scores_inds batch_indsbest_multimask_logitsbest_multimask_iou_scoressinglemask_logitssinglemask_iou_scoresstability_scores is_stablemask_logits_outiou_scores_outs r$rz0SAM2MaskDecoder._dynamic_multimask_via_stabilitys4+111abb!!!QQQ;7-aaae4 <(<"EEE\"6"rs#""""" 33333333XXXXX")XXXvI/I/I/I/I/biI/I/I/I/I/r'