j6V0dZddlmZddlZddlmZddlmZddlZddl m Z ddl m cm Z ddlmcmZddlmZddlmZddlmZmZmZmZmZmZdd lmZd d lmZGd d e jZ Gdde jZ!Gdde jZ"dS)a@ ViTDet backbone adapted from Detectron2. This module implements Vision Transformer (ViT) backbone for object detection. Rope embedding code adopted from: 1. https://github.com/meta-llama/codellama/blob/main/llama/model.py 2. https://github.com/naver-ai/rope-vit 3. https://github.com/lucidrains/rotary-embedding-torch ) annotationsN)partial)Callable)Tensor) PatchEmbed)apply_rotary_enccompute_axial_cisconcat_rel_pos get_abs_poswindow_partitionwindow_unpartition)check_requirements) LayerScalec^eZdZdZ d d!fd Zd"d#dZd$d%dZd&dZd'dZxZ S)( AttentionzIMulti-head Attention block with relative position embeddings and 2d-rope.TFN@dimint num_headsqkv_biasbool use_rel_posrel_pos_zero_init input_sizetuple[int, int] | None cls_tokenuse_rope rope_thetafloat rope_pt_size rope_interpc t||_||z|_|jdz|_||_t j||dz||_t j|||_ ||_ ||_ ||_ | |_ | |_| |_|||||dS)a  Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. qkv_bias (bool): If True, add a learnable bias to query, key, value. use_rel_pos (bool): If True, add relative positional embeddings to the attention map. rel_pos_zero_init (bool): If True, zero initialize relative positional parameters. input_size (tuple[int, int] or None): Input resolution for calculating the relative positional parameter size or rope size. cls_token (bool): Whether a cls_token is present. use_rope (bool): Whether to use rope 2d (independent of use_rel_pos, as it can be used together). rope_theta (float): Control frequencies of rope. rope_pt_size (tuple[int, int] or None): Size of rope in previous stage of training, needed for interpolation or tiling. rope_interp (bool): Whether to interpolate (or extrapolate) rope to match input size. )biasN)super__init__rhead_dimscalernnLinearqkvprojrrrr r"r#_setup_rel_pos_setup_rope_freqs) selfrrrrrrrrr r"r# __class__s g/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/ultralytics/models/sam/sam3/vitdet.pyr)zAttention.__init__,s< "y( ]D( "9S#'999Ic3'' '$  $(& -z::: z*****returnNonec|jsd|_d|_dS|J|jdus Jdt jt jd|dzdz |j|_t jt jd|dzdz |j|_|sLtj |jdtj |jd|\}}t j |dddf}t j |dddf}||z |dz z}| |_ dS) z%Setup relative positional embeddings.NFz not supportedrr{Gz?std)r rel_pos_h rel_pos_wrr, Parametertorchzerosr*init trunc_normal_arangelongrelative_coords)r2rrHWq_coordsk_coordsrFs r4r0zAttention._setup_rel_pos`sO !DN!DN F%%%~&&&&&&ek!jm2Ca2G&W&WXXek!jm2Ca2G&W&WXX  < G ! !$.d ! ; ; ; G ! !$.d ! ; ; ;1<??111d7+<??47+#h.1q59.3355r5c&|js d|_dS|J|j||_tt|j|j|_d}|jr|jd|dz }||d|d|}|j rwtj |jdztj |j }tjtj||dddf}tj||gd }||_dS) zSetup 2d-rope frequencies.N)rtheta?rr)end_xend_y scale_posr9)dtypedevicer)r freqs_cisr"rr r*r compute_cisr#rr@rAfloat32rRpolar ones_likecat)r2rrPrTt cls_freqs_ciss r4r1zAttention._setup_rope_freqsxs4} !DN F%%%   $ *D #  /      =)!,z!} E  "m 'A "K(:(:A>>tQQQwGM =)"rFFscaled_dot_product_attentionviewr/) r2rbsBrGrH_Lrgr.r_r`vs r4forwardzAttention.forwardsT &AAQ 6Q;;JAq!Q6666AADD6Q;;;;gGAq!DIa!e$$ $Ahhqkk!!!Q4>2>>++aAq!,,33A661a1%%1   8! !Q !QA!  $ 4   DAq !T^QUB77A !T^QUB77A *1a 3 3 199q$.!Q33;;Aq!QJJRRSTVWYZ\^__AAq$.!R0088Aq!DDLLQPQSUVVA IIaLLr5) rTFTNFFrNF)rrrrrrrrrrrrrrrrr r!r"rr#r)TN)rrrrr6r7)N)rrr6r7)r6r\rbrr6r) __name__ __module__ __qualname____doc__r)r0r1rarw __classcell__r3s@r4rr)sSS !"&-1#/3!2+2+2+2+2+2+2+h666660%#%#%#%#%#NMMMM........r5rc`eZdZdZdddejejddddddddddfd$fd Zd%d#ZxZ S)&Blockz4Transformer blocks with support of window attention.@TFrNrrr mlp_ratior!rr drop_path norm_layerCallable[..., nn.Module] act_layerrr window_sizerrrr"r#rdropout init_values float | Nonec ttdddlm}m}|||_t||||| | dkr| n| | f| | || |_|rt||ntj |_ |dkr ||ntj |_ |||_||t||z||df|_|rt||ntj |_tj||_| |_dS) a Args: dim (int): Number of input channels. num_heads (int): Number of attention heads in each ViT block. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool): If True, add a learnable bias to query, key, value. drop_path (float): Stochastic depth rate. norm_layer (Callable): Normalization layer constructor. act_layer (Callable): Activation layer constructor. use_rel_pos (bool): If True, add relative positional embeddings to the attention map. rel_pos_zero_init (bool): If True, zero initialize relative positional parameters. window_size (int): Window size for window attention blocks. If it equals 0, then not use window attention. input_size (tuple[int, int] | None): Input resolution for calculating the relative positional parameter size. use_rope (bool): Whether to use rope 2d (independent of use_rel_pos, as it can be used together). rope_pt_size (tuple[int, int] | None): Size of rope in previous stage of training, needed for interpolation or tiling. rope_interp (bool): Whether to interpolate (or extrapolate) rope to match target input size, expected to specify source size as rope_pt_size. cls_token (bool): Whether a cls_token is present. dropout (float): Dropout rate. init_values (float | None): Layer scale init, None for no layer scale. timmr)DropPathMlp) rrrrrrr"r#r)rr) in_featureshidden_featuresrdropN)r(r)r timm.layersrrnorm1rattnrr,Identityls1rnorm2rmlpls2Dropoutrr)r2rrrrrrrrrrrrr"r#rrrrrr3s r4r)zBlock.__init__svV 6"""--------Z__  #/%0A%5%5zzK;U%#     @K]:c{;;;;PRP[P]P]09C),,,R[]]Z__ 3i003     @K]:c{;;;;PRP[P]P]z'** &r5rbrr6c d|}||}|jdkr2|jd|jd}}t||j\}}|||}|jdkrt ||j|||f}||||z}|||| | | |z}|S)z&Forward pass of the transformer block.rrr9) rrrhr rrr rrrrr)r2rbshortcutrGrHpad_hws r4rwz Block.forward&s JJqMM  a  71:qwqzqA(D,<==IAv HHTYYq\\ " "  a  "1d&6AGGA t||DNN1$5$566 6  T^^DHHTXXdjjmm5L5L,M,MNNOO Or5)"rrrrrr!rrrr!rrrrrrrrrrrrrrr"rr#rrrrr!rrrx) ryrzr{r|r, LayerNormGELUr)rwr}r~s@r4rrs>> /1|.0g!"&-1/3!$(%I'I'I'I'I'I'I'Vr5rceZdZdZddddddddd d ejddd dd d d dd dddd d dd d dddfdGfd9 ZedHd>ZdIdBZ dCdCgfdJdFZ xZ S)KViTzThis module implements Vision Transformer (ViT) backbone in :paper:`vitdet`. "Exploring Plain Vision Transformer Backbones for Object Detection", https://arxiv.org/abs/2203.16527. ir&i rTrr)r9r FNimg_sizer patch_sizein_chans embed_dimdepthrrr!rrdrop_path_raterCallable[..., nn.Module] | strrr use_abs_pos tile_abs_posrel_pos_blockstuple[int, ...] | boolrrglobal_att_blockstuple[int, ...]rr" int | Noneuse_interp_ropepretrain_img_sizepretrain_use_cls_tokenretain_cls_tokenrreturn_interm_layersrrln_preln_postbias_patch_embed compile_mode str | Noneuse_act_checkpointc t||_fdt|D} t |_dg|z|_t|tr|r dg|z|_n|D] }!d|j|!< ||_ |j rx|sJt| dks Jdt|jdks Jd|dz}"tj |"tjdd|z|_t| t"r$t%t't| d } t)||f||f||| |_| |_| |_|jr |jsJ|jrB||z||zz}#|r|#dzn|#}$tj tjd|$||_nd |_d tjd| |D}%||_||_tj|_d}&t|D]}!t?d"id|d|d|d|d|%|!d| d| d|j|!d|d|!| vr|ndd||z||zfd|d|||fn||fd|d|j d|d|}'|!| vr|&dz }&||_ |j!|'||_"|r|gt|jzn|g|_#|j&tj$%|jd |r | |ntj&|_'|r | |ntj&|_(|)|j*|Gtj+|j,|d!|_,|j r!|j-rdtj.j/_0d Sd Sd Sd S)#ak Args: img_size (int): Input image size. Only relevant for rel pos or rope. patch_size (int): Patch size. in_chans (int): Number of input image channels. embed_dim (int): Patch embedding dimension. depth (int): Depth of ViT. num_heads (int): Number of attention heads in each ViT block. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool): If True, add a learnable bias to query, key, value. drop_path_rate (float): Stochastic depth rate. norm_layer (Callable or str): Normalization layer constructor or name. act_layer (Callable): Activation layer constructor. use_abs_pos (bool): If True, use absolute positional embeddings. tile_abs_pos (bool): If True, tile absolute positional embeddings instead of interpolation. rel_pos_blocks (tuple[int, ...] | bool): Blocks which have rel pos embeddings. rel_pos_zero_init (bool): If True, zero initialize relative positional parameters. window_size (int): Window size for window attention blocks. global_att_blocks (tuple[int, ...]): Indexes for blocks using global attention (other blocks use window attention). use_rope (bool): Whether to use rope 2d (independent of rel_pos_blocks, as it can be used together). rope_pt_size (int | None): Size of rope in previous stage of training, needed for interpolation or tiling. use_interp_rope (bool): Whether to interpolate (or extrapolate) rope to match target input size, expected to specify source size as rope_pt_size. pretrain_img_size (int): Input image size for pretraining models. pretrain_use_cls_token (bool): If True, pretraining models use class token. retain_cls_token (bool): Whether cls_token should be retained. dropout (float): Dropout rate. Applied in residual blocks of attn, mlp and inside the mlp. return_interm_layers (bool): Whether to return intermediate layers (all global attention blocks). init_values (float | None): Layer scale init, None for no layer scale. ln_pre (bool): If True, apply layer norm before transformer blocks. ln_post (bool): If True, apply layer norm after transformer blocks. bias_patch_embed (bool): If True, use bias in conv for patch embed. compile_mode (str | None): Mode to compile the forward, or None to disable. use_act_checkpoint (bool): If True, use activation checkpointing. cg|]}|v| Sr).0irs r4 z ViT.__init__..s$VVVa1DU;U;U;U;U;Ur5FTrz&windowing not supported with cls tokenz$rel pos not supported with cls tokenr%rgh㈵>)eps) kernel_sizestriderrr'Nc6g|]}|Sr)item)rrbs r4rz ViT.__init__..s JJJAqvvxxJJJr5rrrrrrrrrrrrr"r#rrrr:r;)mode fullgraphr)1r(r)rrangelist full_attn_idsr isinstancerrlensumr,r?r@randnclass_embeddingstrrgetattrr patch_embedrrrA pos_embedlinspacerr ModuleListblocksrrappendr channel_listrBrCrrrapply _init_weightscompilerwtraining_dynamoconfig optimize_ddp))r2rrrrrrrrrrrrrrrrrrr"rrrrrrrrrrrrwindow_block_indexesrr+ num_patches num_positionsdpr cur_stageblockr3s) ` r4r)z ViT.__init__?sL &<#VVVV5<<VVV!"344$go nd + + . .#'&5.D  # . .)-#A&& 0   V) ) ))+,,1113[111t*++q0002X000tOE#%< Aq)8T8T0T#U#UD j# & & D Z!8!8dCCCJ%#Z0 +!    )&   $# # ##   ", :?PT^?^_K1GX[1__[M\%+a *R*RSSDNN!DNKJ>5!I!IJJJ$&moo  u & &AI#)$)"  a&&  &: $)!/22#4"3,-0D+D+DKK!% 2H 4JKK"=I % G ! !$.d ! ; ; ;/5Hjj+++2;== 07Jzz),,,R[]]  4%&&&  # =LTXYYYDL& :4= :49 $111 $ # : : : :r5m nn.Moduler6r7ct|tjrrtj|jdt|tjr.|j)tj|jddSdSdSt|tjrLtj|jdtj|jddSdS)zInitialize the weights.r:r;NrrM) rr,r-rBrCweightr' constant_r)rs r4rzViT._init_weightss a # # - G ! !!( ! 5 5 5!RY'' -AF,>!!!&!,,,,, - -,>,> 2< ( ( - G  afa ( ( ( G  ah , , , , , - -r5rb torch.Tensorlist[torch.Tensor]c ||}|jd|jd}}d}|jr3tj|j|ddgd}d}|j-|t|j|j ||f|j|j z}| |}g}t|j D]7\}}|jr|jrt!j||d}n ||}||jd ks|jr||jvr||jd kr||}|dd|df}|jd kr|dd dd}nr|jd ksJt-j|jdx}}||jd|||jd dd dd}||9|S) z&Vit forward path and get feature maps.rr9rrSN)tilingF) use_reentrantrerdr&)rrhrr@rYrrnrr rrr enumeraterrr checkpointrrrrgrlrirjrkr) r2rbhwrroutputsrblkfeatss r4rwz ViT.forwards   Q  wqz171:1     4/1aAqIIIAA > %K+A%( A KKNN ,, & &FAs& 4= )#qFFFCFFT'+++1J+qTXTfOfOf*2... QA!!!QRR%:??!MM!Q155EE :???? Iek!n555A!MM%+a.!Q BPPXXYZ\]_`bcddEu%%%r5iimgsz list[int]c|jD]}|jdkr|j|d|jz|d|jzf|j|d|jz|d|jzfdS)zCSetup rel pos embeddings and rope freqs for a new input image size.rr)rN)rrrr0rr1)r2rrs r4 set_imgszz ViT.set_imgszs[ p pE A%% J % %%(do2MuUVx[_[jOj1k % l l l J ( (U1X5PRWXYRZ^b^mRm4n ( o o o o  p pr5)>rrrrrrrrrrrrrr!rrrr!rrrrrrrrrrrrrrrrrrr"rrrrrrrrrrr!rrrrrrrrrrrrrr)rrr6r7)rbrr6r)rr) ryrzr{r|r,rr) staticmethodrrwrr}r~s@r4rr:s  #5@.0g !1>"&-:#' %!$'+!%%*$(!%#'#'Af:f:f:f:f:f:f:P---\-++++Z-1$<pppppppppr5r)#r| __future__rri functoolsrtypingrr@torch.nnr,torch.nn.functional functionalrotorch.utils.checkpointutilsrr%ultralytics.models.sam.modules.blocksr$ultralytics.models.sam.modules.utilsrr r r r r ultralytics.utils.checksr model_miscrModulerrrrr5r4rs #"""""  +++++++++<<<<<<877777""""""lllll lll^_____BI___Dkpkpkpkpkp")kpkpkpkpkpr5