/j#dZddlmZddlZddlmZddlmcmZddl m Z ddl m Z m Z mZmZmZmZddlmZdZGd d ejZGd d ejZGd dejZGddejZGddejZGddejZGddejZGddejZGddejZGddejZGddeZ Gdd ejZ!Gd!d"eZ"Gd#d$eZ#Gd%d&ejZ$Gd'd(ejZ%Gd)d*ejZ&Gd+d,ejZ'Gd-d.ejZ(Gd/d0ejZ)Gd1d2ejZ*Gd3d4ejZ+Gd5d6ejZ,Gd7d8ejZ-Gd9d:e%Z.Gd;dejZ0Gd?d@e0Z1GdAdBejZ2GdCdDejZ3GdEdFejZ4GdGdHejZ5GdIdJejZ6GdKdLejZ7GdMdNeZ8GdOdPeZ9GdQdRejjZ:GdSdTejZ;GdUdVeZ<GdWdXejZ=GdYdZejZ>Gd[d\ejZ?Gd]d^ejZ@Gd_d`eZAGdadbejZBGdcddejZCGdedfejZDGdgdhejZEGdidjejZFGdkdlejZGGdmdnejZHGdodpejZIGdqdreZJGdsdtejZKdS)uzBlock modules.) annotationsN)fuse_conv_and_bn)ConvDWConv GhostConv LightConvRepConvautopad)TransformerBlock)'C1C2C2PSAC3C3TRCIBDFLELAN1PSASPPSPPELANSPPFAConvADown AttentionBNContrastiveHead Bottleneck BottleneckCSPC2fC2fAttnC2fCIBC2fPSAC3GhostC3k2C3xCBFuseCBLinearContrastiveHeadGhostBottleneckHGBlockHGStemImagePoolingAttnProtoRepC3 RepNCSPELAN4RepVGGDW ResNetLayerSCDown TorchVisionc.eZdZdZd d fd Zd d ZxZS) rzIntegral module of Distribution Focal Loss (DFL). Proposed in Generalized Focal Loss https://ieeexplore.ieee.org/document/9792391 c1intcrttj|dddd|_t j|t j}tj | d|dd|jj j dd<||_ dS)zInitialize a convolutional layer with a given number of input channels. Args: c1 (int): Number of input channels. rFbiasdtypeN)super__init__nnConv2drequires_grad_convtorcharangefloat Parameterviewweightdatar6)selfr6x __class__s a/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/ultralytics/nn/modules/block.pyr>z DFL.__init__@s Ib!QU333BB5II L5; / / /#%<q"a0C0C#D#D aaa rK torch.Tensorreturnc|j\}}}|||d|j|ddd|d|S)zCApply the DFL module to input tensor and return transformed output.r)shaperBrGr6 transposesoftmax)rJrKb_as rMforwardz DFL.forwardLsh'1ayy1dgq11;;AqAAII!LLMMRRSTVWYZ[[[rN)r5)r6r7rKrOrPrO__name__ __module__ __qualname____doc__r>rZ __classcell__rLs@rMrr:sh       \\\\\\\\rNrc.eZdZdZd dfd Zdd ZxZS)r-zBUltralytics YOLO models mask Proto module for segmentation models. r6r7c_c2c tt||d|_t j||dddd|_t||d|_t|||_dS)aInitialize the Ultralytics YOLO models mask Proto module with specified number of protos and masks. Args: c1 (int): Input channels. c_ (int): Intermediate channels. c2 (int): Output channels (number of protos). krSrTr9N) r=r>rcv1r?ConvTranspose2dupsamplecv2cv3)rJr6rfrgrLs rMr>zProto.__init__Vsy B!$$$*2r1aFFF B!$$$B<<rNrKrOrPc |||||S)zEPerform a forward pass through layers using an upsampled input image.)rprornrlrJrKs rMrZz Proto.forwardds6xxtxx{{!;!;<<===rN)rdre)r6r7rfr7rgr7r[r\rbs@rMr-r-Ss\LL        >>>>>>>>rNr-c,eZdZdZd fd Zd d ZxZS) r+zStemBlock of PPHGNetV2 with 5 convolutions and one maxpool2d. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/backbones/hgnet_v2.py r6r7cmrgc <tt||ddtj|_t||dzdddtj|_t|dz|dddtj|_t|dz|ddtj|_t||ddtj|_ tj dddd|_ dS) zInitialize the StemBlock of PPHGNetV2. Args: c1 (int): Input channels. cm (int): Middle channels. c2 (int): Output channels. rirSactrrT) kernel_sizestridepadding ceil_modeN) r=r>rr?ReLUstem1stem2astem2bstem3stem4 MaxPool2dpool)rJr6rtrgrLs rMr>zHGStem.__init__os "b!QBGII666 2rQw1aRWYY??? 27B1aRWYY??? "q&"a ::: "b!QBGII666 LQq!tTTT rNrKrOrPc||}tj|gd}||}tj|gd}||}||}t j||gd}||}| |}|S)+Forward pass of a PPHGNetV2 backbone layer.)rrrrrdim) r}Fpadr~rrrCcatrr)rJrKx2x1s rMrZzHGStem.forwards JJqMM E!\\\ " " [[^^ U2||| $ $ [[__ YYq\\ Ir2hA & & & JJqMM JJqMMrN)r6r7rtr7rgr7r[r\rbs@rMr+r+isa UUUUUU        rNr+cReZdZdZddddejfdfd ZddZxZS)r*zHG_Block of PPHGNetV2 with 2 convolutions and LightConv. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/backbones/hgnet_v2.py riFr6r7rtrgrkn lightconvboolshortcutrw nn.Modulec t t|rtnt t j fdt |D|_t|zz|dzdd|_t|dz|dd|_ |o|k|_ dS)aInitialize HGBlock with specified parameters. Args: c1 (int): Input channels. cm (int): Middle channels. c2 (int): Output channels. k (int): Kernel size. n (int): Number of LightConv or Conv blocks. lightconv (bool): Whether to use LightConv. shortcut (bool): Whether to use shortcut connection. act (nn.Module): Activation function. c3FK|]}|dkrnVdS)rrkrwN).0irwblockr6rtrks rM z#HGBlock.__init__..sC__QRuu166RRr2LLL______rNrSrrvN) r=r>r rr? ModuleListrangemscecadd) rJr6rtrgrkrrrrwrrLs `` ` `@rMr>zHGBlock.__init__s. &0 D________V[\]V^V^_____rAF{B!GQs;;;rQwAqc222(brNrKrOrPc|gfd|jD||t jd|jr|znS)rc3:K|]}|dVdSNrrrys rMrz"HGBlock.forward../**a1R5******rNr)extendrrrrCrrrJrKrs @rMrZzHGBlock.forwardso C ****46****** GGDGGEIaOO,, - -'q1uua'rN)r6r7rtr7rgr7rkr7rr7rrrrrwrr[) r]r^r_r`r?r|r>rZrarbs@rMr*r*sy )))))))<((((((((rNr*c.eZdZdZd dfd Zdd ZxZS)rzDSpatial Pyramid Pooling (SPP) layer https://arxiv.org/abs/1406.4729. r6r7rgrktuple[int, ...]ct|dz}t||dd|_t|t |dzz|dd|_t jd|D|_dS)zInitialize the SPP layer with input/output channels and pooling kernel sizes. Args: c1 (int): Input channels. c2 (int): Output channels. k (tuple): Kernel sizes for max pooling. rSrcBg|]}tj|d|dzS)rrSrxryrz)r?r)rrKs rM z SPP.__init__..s/aaaZ[ 1aSTf U U UaaarNN) r=r>rrllenror?rr)rJr6rgrkrfrLs rMr>z SPP.__init__s  1WB1%%c!ffqj)2q!44aa_`aaabbrNrKrOrPc||tjgfd|jDzdS)zBForward pass of the SPP layer, performing spatial pyramid pooling.c&g|] }|Srr)rrrKs rMrzSPP.forward..s!(>(>(>!1(>(>(>rNr)rlrorCrrrrs `rMrZz SPP.forwardsN HHQKKxx 1#(>(>(>(>tv(>(>(>">BBCCCrN)r)r6r7rgr7rkrr[r\rbs@rMrrskNN c c c c c c cDDDDDDDDrNrc.eZdZdZddfd ZddZxZS)rzGSpatial Pyramid Pooling - Fast (SPPF) layer for YOLOv5 by Glenn Jocher.rriFr6r7rgrkrrrc$t|dz}t||ddd|_t||dzz|dd|_t j|d|dz|_||_|o||k|_ dS)aInitialize the SPPF layer with given input/output channels and kernel size. Args: c1 (int): Input channels. c2 (int): Output channels. k (int): Kernel size. n (int): Number of pooling iterations. shortcut (bool): Whether to use shortcut connection. Notes: This module is equivalent to SPP(k=(5, 9, 13)). rSrFrvrN) r=r>rrlror?rrrr)rJr6rgrkrrrfrLs rMr>z SPPF.__init__s  1WB1%000a!e b!Q//!AqAvFFF(brNrKrOrPc .|gfdttddDt jdtddr|znS)zRApply sequential pooling operations to input and return concatenated feature maps.c3NK|]}dV dSrr)rrXrJrs rMrzSPPF.forward..s1EE1"EEEEEErNrrirrF)rlrrgetattrrorCrrs` @rMrZz SPPF.forwards XXa[[M EEEEEgdC.C.C(D(DEEEEEE HHUYq!__ % %eU33:q1uu:rN)rriF) r6r7rgr7rkr7rr7rrr[r\rbs@rMrrs\QQ)))))))*;;;;;;;;rNrc.eZdZdZd d fd Zdd ZxZS)r z"CSP Bottleneck with 1 convolution.rr6r7rgrctt|dd|_t jfdt |D|_dS)zInitialize the CSP Bottleneck with 1 convolution. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of convolutions. rc3:K|]}tdVdS)riNr)rrXrgs rMrzC1.__init__..s- C CQb"a C C C C C CrNN)r=r>rrlr? Sequentialrr)rJr6rgrrLs ` rMr>z C1.__init__s^ B1%% C C C C%(( C C CDrNrKrOrPc\||}|||zS)z:Apply convolution and residual connection to input tensor.)rlrrs rMrZz C1.forwards% HHQKKvvayy1}rNr)r6r7rgr7rr7r[r\rbs@rMr r sc,, E E E E E E ErNr c.eZdZdZddfd ZddZxZS)rz#CSP Bottleneck with 2 convolutions.rT?r6r7rgrrrgerEcRtt||z_t |djzdd_t djz|d_tjfdt|D_ dS)a_Initialize a CSP Bottleneck with 2 convolutions. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of Bottleneck blocks. shortcut (bool): Whether to use shortcut connections. g (int): Groups for convolutions. e (float): Expansion ratio. rSrc 3VK|]#}tjjddV$dS)ririr?rkrNrcrrXrrJrs rMrzC2.__init__..s? v vhiDFDFHaK[_b!c!c!c v v v v v vrNN r=r>r7rrrlror?rrrrJr6rgrrrrrLs` `` rMr>z C2.__init__s R!VAJ1--DF B** v v v v v vmrstmumu v v vwrNrKrOrPc||dd\}}|tj|||fdS)z.2?ttfgz$&$&(AIY]`aaattttttrNN) r=r>r7rrrlror?rrrrs` `` rMr>z C2f.__init__#s R!VAJ1--Q$&("a00ttttttkpqrkskstttttrNrKrOrPc t||ddfd|jD|t jdS)zForward pass through C2f layer.rSrc3:K|]}|dVdSrrrs rMrzC2f.forward..7rrN)listrlrrrrorCrrs @rMrZz C2f.forward4so !""1a(( ) ) ****46******xx !Q(((rNc*|||j|jfdddgfd|jD|t jdS).Forward pass using split() instead of chunk().rrc3:K|]}|dVdSrrrs rMrz$C2f.forward_split..>rrN)rlsplitrrrrorCrrs @rM forward_splitzC2f.forward_split:s HHQKK  tvtv. 2 2 qT1Q4L ****46******xx !Q(((rNrFrrrr[r]r^r_r`r>rZrrarbs@rMrr swFFuuuuuuu")))) ))))))))rNrc.eZdZdZddfd ZddZxZS)rz#CSP Bottleneck with 3 convolutions.rTrr6r7rgrrrrrrEc\tt||zt|dd|_t|dd|_tdz|d|_tjfdt|D|_ dS)aaInitialize the CSP Bottleneck with 3 convolutions. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of Bottleneck blocks. shortcut (bool): Whether to use shortcut connections. g (int): Groups for convolutions. e (float): Expansion ratio. rrSc 3BK|]}tddVdS)))rrrrrNrrrXrfrrs rMrzC3.__init__..Us; n n`aBHaCSWZ![![![ n n n n n nrNN) r=r>r7rrlrorpr?rrr rJr6rgrrrrrfrLs `` @rMr>z C3.__init__Es  a[[B1%%B1%%BA&& n n n n n nejklemem n n norNrKrOrPc |tj|||||fdS)z.ls? v vhiDGTWhM]ab!c!c!c v v v v v vrNN)r=r>r7rfr?rrrrs` `` rMr>z C3x.__init___sr RHa333b1f++ v v v v v vmrstmumu v v vwrNrrr]r^r_r`r>rarbs@rMr%r%\sS,, x x x x x x x x x x xrNr%c.eZdZdZddfd ZddZxZS)r.zRep C3.rirr6r7rgrrrEctt||zt|dd|_t|dd|_t jfdt|D|_ |krt|ddnt j |_ dS)zInitialize RepC3 module with RepConv blocks. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of RepConv blocks. e (float): Expansion ratio. rc0g|]}tSr)r rrXrfs rMrz"RepC3.__init__..s! C C CQR C C CrNN) r=r>r7rrlror?rrrIdentityrp)rJr6rgrrrfrLs @rMr>zRepC3.__init__rs  a[[B1%%B1%% C C C C%(( C C CD)+r4B1%%%r{}}rNrKrOrPc||||||zS)zForward pass of RepC3 module.)rprrlrorrs rMrZz RepC3.forwards9xxtxx{{++dhhqkk9:::rN)rirr6r7rgr7rr7rrEr[r\rbs@rMr.r.osbMEEEEEEE ;;;;;;;;rNr.c&eZdZdZddfd ZxZS)rz"C3 module with TransformerBlock().rTrr6r7rgrrrrrrEct||||||t||z}t||d||_dS)a[Initialize C3 module with TransformerBlock. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of Transformer blocks. shortcut (bool): Whether to use shortcut connections. g (int): Groups for convolutions. e (float): Expansion ratio. rRN)r=r>r7r rrs rMr>z C3TR.__init__sO RHa333 a[[!"b!Q//rNrrrrbs@rMrrsH,, 0 0 0 0 0 0 0 0 0 0 0rNrc&eZdZdZddfd ZxZS)r#z!C3 module with GhostBottleneck().rTrr6r7rgrrrrrrEct||||||t||ztjfdt |D|_dS)a_Initialize C3 module with GhostBottleneck. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of Ghost bottleneck blocks. shortcut (bool): Whether to use shortcut connections. g (int): Groups for convolutions. e (float): Expansion ratio. c38K|]}tVdS)N)r)rs rMrz#C3Ghost.__init__..s- K KQR!8!8 K K K K K KrNNr=r>r7r?rrrrs @rMr>zC3Ghost.__init__sd RHa333 a[[ K K K K%(( K K KLrNrrrrbs@rMr#r#sS++ M M M M M M M M M M MrNr#c.eZdZdZddfd Zdd ZxZS)r)zGGhost Bottleneck https://github.com/huawei-noah/Efficient-AI-Backbones.rirr6r7rgrksc t|dz}tjt ||dd|dkrt ||||dntjt ||ddd|_|dkr9tjt ||||dt||dddntj|_ dS)zInitialize Ghost Bottleneck module. Args: c1 (int): Input channels. c2 (int): Output channels. k (int): Kernel size. s (int): Stride. rSrFrvN) r=r>r?rrrrrBrr)rJr6rgrkrrfrLs rMr>zGhostBottleneck.__init__s  1WM b"a # #/0AvvF2r1aU + + + +2;== b"a . . .   ^_bc]c]cBM&RA59994B1RW;X;X;X Y Y Yikitiviv rNrKrOrPcX||||zS)z3Apply skip connection and addition to input tensor.)rBrrrs rMrZzGhostBottleneck.forwards#yy||dmmA....rNrr6r7rgr7rkr7rr7r[r\rbs@rMr)r)s\QQ       (////////rNr)c0eZdZdZ ddfd ZddZxZS)rzStandard bottleneck.Trrrr6r7rgrrrrktuple[int, int]rrEctt||z}t|||dd|_t|||dd||_|o||k|_dS)aZInitialize a standard bottleneck module. Args: c1 (int): Input channels. c2 (int): Output channels. shortcut (bool): Whether to use shortcut connection. g (int): Groups for convolutions. k (tuple): Kernel sizes for convolutions. e (float): Expansion ratio. rrrN)r=r>r7rrlror rJr6rgrrrkrrfrLs rMr>zBottleneck.__init__su  a[[B!a((B!a1---(brNrKrOrPc|jr+||||zn'|||S)z3Apply bottleneck with optional shortcut connection.)rrorlrrs rMrZzBottleneck.forwardsE,0HOq488DHHQKK(((($((488A;;:O:OOrNTrrr r6r7rgr7rrrr7rkrrrEr[r\rbs@rMrrsklo)))))))&PPPPPPPPrNrc.eZdZdZddfd ZddZxZS)rzGCSP Bottleneck https://github.com/WongKinYiu/CrossStagePartialNetworks.rTrr6r7rgrrrrrrEctt||zt|dd|_t j|ddd|_t jddd|_tdz|dd|_ t j dz|_ t j |_ t jfdt|D|_dS)aIInitialize CSP Bottleneck. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of Bottleneck blocks. shortcut (bool): Whether to use shortcut connections. g (int): Groups for convolutions. e (float): Expansion ratio. rFr9rSc3@K|]}tdVdSrrNrrs rMrz)BottleneckCSP.__init__..s6 Z ZABHa3!G!G!G Z Z Z Z Z ZrNN)r=r>r7rrlr?r@rorpcv4 BatchNorm2dbnSiLUrwrrrrs `` @rMr>zBottleneckCSP.__init__s  a[[B1%%9RQ6669RQ666BAq)).R((799 Z Z Z Z Z ZQVWXQYQY Z Z Z[rNrKrOrPc B||||}||}|||tj||fdS)z)Apply CSP bottleneck with 4 convolutions.r) rprrlrorrwrrCr)rJrKy1y2s rMrZzBottleneckCSP.forwardsr XXdffTXXa[[)) * * XXa[[xxB8Q)?)?!@!@AABBBrNrrr[r\rbs@rMrrskQQ\\\\\\\*CCCCCCCCrNrc.eZdZdZddfd Zdd ZxZS) ResNetBlockz.ResNet block with standard convolution layers.rrRr6r7rgrrc |t||z}t||ddd|_t||d|dd|_t||dd|_|dks||kr&t jt||d|dnt j|_ dS) zInitialize ResNet block. Args: c1 (int): Input channels. c2 (int): Output channels. s (int): Stride. e (int): Expansion ratio. rTrkrrwrirkrprwFrN) r=r>rrlrorpr?rrr)rJr6rgrrc3rLs rMr>zResNetBlock.__init__s  VB!qd333B!qA4888B!///LMQRFFVX\^V^V^ d2rQ!&G&G&GHHHdfdodqdq rNrKrOrPc tj||||||zS)z&Forward pass through the ResNet block.)rrelurprorlrrrs rMrZzResNetBlock.forwardsEvdhhtxx 4455 a8H8HHIIIrN)rrR)r6r7rgr7rr7rr7r[r\rbs@rMrrsk88rrrrrrr JJJJJJJJrNrc.eZdZdZddfd ZddZxZS)r1z)ResNet layer with multiple ResNet blocks.rFrRr6r7rgris_firstrrrc t||_|jrDtjt |ddddtjddd|_d St||g}| fd t|dz Dtj||_d S) a,Initialize ResNet layer. Args: c1 (int): Input channels. c2 (int): Output channels. s (int): Stride. is_first (bool): Whether this is the first layer. n (int): Number of ResNet blocks. e (int): Expansion ratio. rSriTrrrrc<g|]}tzdS)rr)r)rrXrgrs rMrz(ResNetLayer.__init__..2s.QQQq;q2vr1:::QQQrNN) r=r>r$r?rrrlayerrrr) rJr6rgrr$rrblocksrLs ` ` rMr>zResNetLayer.__init__s   = 0RqA555r|PQZ[ef7g7g7gDJJJ""b!q1112F MMQQQQQE!a%LLQQQ R R R/DJJJrNrKrOrPc,||S)z&Forward pass through the ResNet layer.)r(rrs rMrZzResNetLayer.forward5szz!}}rN)rFrrR) r6r7rgr7rr7r$rrr7rr7r[r\rbs@rMr1r1s\330000000.rNr1c.eZdZdZddfd ZddZxZS)MaxSigmoidAttnBlockzMax Sigmoid attention block.rFr6r7rgnhrgcscalerct||_||z|_||krt ||ddnd|_t j|||_t j tj ||_ t ||ddd|_ |r)t j tjd|ddnd|_dS)a?Initialize MaxSigmoidAttnBlock. Args: c1 (int): Input channels. c2 (int): Output channels. nh (int): Number of heads. ec (int): Embedding channels. gc (int): Guide channels. scale (bool): Whether to use learnable scale parameter. rFrNrirr)r=r>r/hcrrr?LinearglrFrCzerosr: proj_convonesr1)rJr6rgr/rr0r1rLs rMr>zMaxSigmoidAttnBlock.__init__=s (24(($r2....)B##LR11 b"QE:::>CLR\%*QAq"9"9::: rNrKrOguiderPc|j\}}}}||}|||jd|j|j}|j||n|}|||j|j||}t jd||}|dd}||jdzz }||j dddddfz}| |j z}| |}|||jd||}|| dz}||d||S) zForward pass of MaxSigmoidAttnBlock. Args: x (torch.Tensor): Input tensor. guide (torch.Tensor): Guide tensor. Returns: (torch.Tensor): Output tensor after attention. rNzbmchw,bnmc->bmhwnrrrrrS)rTr5rGr/r3rrCeinsummaxr:sigmoidr1r7 unsqueeze) rJrKr9bsrXhwembedaws rMrZzMaxSigmoidAttnBlock.forwardQs>g Aq! 2u{1~tw@@"g1 q 2twA66 \-ue < < VVV^^A  47C<  $)D!!!T4/0 0 ZZ\\DJ & NN1   FF2twAq ) )  Q vvb"a###rN)rr-r.F) r6r7rgr7r/r7rr7r0r7r1rrKrOr9rOrPrOr\rbs@rMr,r,:sc&&MMMMMMM($$$$$$$$rNr,cDeZdZdZ ddfd ZddZddZxZS)r z*C2f module with an additional attn module.rr-r.Frr6r7rgrrr/r0rrrrrEc tt|| z_t |djzdd_t d|zjz|d_tjfdt|D_ tjj|||_ dS)aInitialize C2f module with attention mechanism. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of Bottleneck blocks. ec (int): Embedding channels for attention. nh (int): Number of heads for attention. gc (int): Guide channels for attention. shortcut (bool): Whether to use shortcut connections. g (int): Groups for convolutions. e (float): Expansion ratio. rSrric 3VK|]#}tjjddV$dSrrrs rMrz#C2fAttn.__init__..rrN)r0rr/N) r=r>r7rrrlror?rrrr,attn) rJr6rgrrr/r0rrrrLs ` `` rMr>zC2fAttn.__init__qs2 R!VAJ1--Q$&("a00ttttttkpqrksksttttt'2"LLL rNrKrOr9rPcjt||ddfd|jD|d||tj dS)zForward pass through C2f layer with attention. Args: x (torch.Tensor): Input tensor. guide (torch.Tensor): Guide tensor for attention. Returns: (torch.Tensor): Output tensor after processing. rSrc3:K|]}|dVdSrrrs rMrz"C2fAttn.forward..rrNr) rrlrrrappendrHrorCrrJrKr9rs @rMrZzC2fAttn.forwards !""1a(( ) ) ****46****** 1R5%(()))xx !Q(((rNct|||j|jfdfd|jD|d||tj dS)zForward pass using split() instead of chunk(). Args: x (torch.Tensor): Input tensor. guide (torch.Tensor): Guide tensor for attention. Returns: (torch.Tensor): Output tensor after processing. rc3:K|]}|dVdSrrrs rMrz(C2fAttn.forward_split..rrNr) rrlrrrrrKrHrorCrrLs @rMrzC2fAttn.forward_splits !""DFDF#3Q77 8 8 ****46****** 1R5%(()))xx !Q(((rN)rr-rr.Frr)r6r7rgr7rr7rr7r/r7r0r7rrrr7rrErDrrbs@rMr r ns44 MMMMMMM@ ) ) ) ) ) ) ) ) ) ) ) )rNr c0eZdZdZ ddfd ZddZxZS)r,zKImagePoolingAttn: Enhance the text embeddings with image-aware information.rdrr.riFrr7chrctr/rkr1rcvtt|}tjtj|tj||_tjtjtj|_tjtjtj|_ tj||_ |r)tj tj dgdnd|_tjfd|D|_tjfdt#|D|_|_||_||_|z|_|_dS)aInitialize ImagePoolingAttn module. Args: ec (int): Embedding channels. ch (tuple): Channel dimensions for feature maps. ct (int): Channel dimension for text embeddings. nh (int): Number of attention heads. k (int): Kernel size for pooling. scale (bool): Whether to use learnable scale parameter. gT requires_gradrc>g|]}tj|dS)r)rx)r?r@)r in_channelsrs rMrz-ImagePoolingAttn.__init__..s,)j)j)jXc")KQR*S*S*S)j)j)jrNc<g|]}tjfSr)r?AdaptiveMaxPool2d)rrXrks rMrz-ImagePoolingAttn.__init__..s(&W&W&Wr';QF'C'C&W&W&WrNN)r=r>rr?r LayerNormr4querykeyvalueprojrFrCtensorr1r projectionsrim_poolsrr/nfr3rk) rJrrQrRr/rkr1rbrLs ` ` rMr>zImagePoolingAttn.__init__sc  WW]2<#3#3RYr25F5FGG =b!1!129R3D3DEE]2<#3#3RYr25F5FGG Ib"%% NS\R\%,u"5"5TJJJJY\ =)j)j)j)jgi)j)j)jkk &W&W&W&WUSUYY&W&W&WXX (rNrKlist[torch.Tensor]textrOrPcx|djdt||jksJ|jdzfdt ||j|jD}tj|d dd}| |}| |}| |}| d|j|j}| d|j|j}| d|j|j}tjd||}||jdzz }t#j|d}tjd ||}|| d|j}||jz|zS) zForward pass of ImagePoolingAttn. Args: x (list[torch.Tensor]): List of input feature maps. text (torch.Tensor): Text embeddings. Returns: (torch.Tensor): Enhanced text embeddings. rrScjg|]/\}}}|||d0S)r)rG)rrKr^rr? num_patchess rMrz,ImagePoolingAttn.forward..sA t t t!T4TT$$q'']]  B 4 4 t t trNrrrzbnmc,bkmc->bmnkrzbmnk,bkmc->bnmc)rTrrbrkzipr`rarCrrUr[r\r]reshaper/r3r;rrVr^rr1) rJrKrdqrkvrCr?rgs @@rMrZzImagePoolingAttn.forwardsqTZ]1vv    fai t t t t tCPQSWSceierLsLs t t t IaR * *1a 0 0 JJt   HHQKK JJqMM IIb"dgtw / / IIb"dgtw / / IIb"dgtw / / \+Q 2 2 47C<  Yrr " " " L*B 2 2 IIaiiB00 1 14:~$$rN)rdrr.rPriF) rr7rQrrRr7r/r7rkr7r1r)rKrcrdrOrPrOr\rbs@rMr,r,scUUns:%%%%%%%%rNr,c*eZdZdZfdZddZxZS) r(zZImplements contrastive learning head for region-text similarity in vision-language models.cDttjt jdg|_tjt jgt jdz|_ dS)zBInitialize ContrastiveHead with region-text similarity parameters.$g$I$I,@N) r=r>r?rFrCr_r:r8log logit_scalerJrLs rMr>zContrastiveHead.__init__sq Lug!6!677 < 2h9O9O9S9S9U9U(UVVrNrKrOrArPctj|dd}tj|dd}tjd||}||jz|jzS)zForward function of contrastive learning. Args: x (torch.Tensor): Image features. w (torch.Tensor): Text features. Returns: (torch.Tensor): Similarity scores. rrSrrrbchw,bkc->bkhw)r normalizerCr;rpexpr:rJrKrAs rMrZzContrastiveHead.forwardse KqA & & & KrQ ' ' ' L)1a 0 04#'')))DI55rNrKrOrArOrPrOr\rbs@rMr(r(sWddWWWWW 6 6 6 6 6 6 6 6rNr(cJeZdZdZd fd ZdZed d Zd d ZxZ S)rzBatch Norm Contrastive Head using batch norm instead of l2-normalization. Args: embed_dims (int): Embed dimensions of text and image features. embed_dimsr7c.ttj||_tjt jdg|_tjdt j gz|_ dS)zvInitialize BNContrastiveHead. Args: embed_dims (int): Embedding dimensions for features. rngN) r=r>r?rnormrFrCr_r:r8rp)rJrzrLs rMr>zBNContrastiveHead.__init__sn N:.. Lug!6!677 <uz"~~(=>>rNc*|`|`|`|j|_dS)zCFuse the batch normalization layer in the BNContrastiveHead module.N)r|r:rp forward_fuserZrJs rMfusezBNContrastiveHead.fuses I I  ( rNrKrOrArPc|S)z5Passes image features through unchanged after fusing.r)rKrAs rMr~zBNContrastiveHead.forward_fuse&s rNc||}tj|dd}tjd||}||jz|jzS)zForward function of contrastive learning with batch normalization. Args: x (torch.Tensor): Image features. w (torch.Tensor): Text features. Returns: (torch.Tensor): Similarity scores. rrSrsrt)r|rrurCr;rprvr:rws rMrZzBNContrastiveHead.forward+s^ IIaLL KrQ ' ' ' L)1a 0 04#'')))DI55rN)rzr7rx) r]r^r_r`r>r staticmethodr~rZrarbs@rMrr s ? ? ? ? ? ?)))\66666666rNrc(eZdZdZ ddfd ZxZS) RepBottleneckzRep bottleneck.Trrrr6r7rgrrrrkrrrEct||||||t||z}t|||dd|_dS)aKInitialize RepBottleneck. Args: c1 (int): Input channels. c2 (int): Output channels. shortcut (bool): Whether to use shortcut connection. g (int): Groups for convolutions. k (tuple): Kernel sizes for convolutions. e (float): Expansion ratio. rrN)r=r>r7r rlr s rMr>zRepBottleneck.__init__?sS R1a333 a[[2r1Q4++rNr r rrbs@rMrr<sOlo,,,,,,,,,,,rNrc&eZdZdZddfd ZxZS)RepCSPzXRepeatable Cross Stage Partial Network (RepCSP) module for efficient feature extraction.rTrr6r7rgrrrrrrEct||||t||ztjfdt |D|_dS)aJInitialize RepCSP layer. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of RepBottleneck blocks. shortcut (bool): Whether to use shortcut connections. g (int): Groups for convolutions. e (float): Expansion ratio. c3@K|]}tdVdSr)rrs rMrz"RepCSP.__init__..as6 ] ]qr2xc!J!J!J ] ] ] ] ] ]rNNrrs `` @rMr>zRepCSP.__init__Tsp RHa333 a[[ ] ] ] ] ] ]TYZ[T\T\ ] ] ]^rNrrrrbs@rMrrQsSbb _ _ _ _ _ _ _ _ _ _ _rNrc6eZdZdZddfd Zdd ZddZxZS)r/z CSP-ELAN.rr6r7rgr c4rc t|dz|_t||dd|_t jt|dz||t||dd|_t jt|||t||dd|_ t|d|zz|dd|_ dS)aInitialize CSP-ELAN layer. Args: c1 (int): Input channels. c2 (int): Output channels. c3 (int): Intermediate channels. c4 (int): Intermediate channels for RepCSP. n (int): Number of RepCSP blocks. rSrriN) r=r>rrrlr?rrrorpr)rJr6rgr rrrLs rMr>zRepNCSPELAN4.__init__gs qB1%%=aQ!7!7b"a9K9KLL=B!2!2DRA4F4FGGa"f r1a00rNrKrOrPct||ddfd|j|jfD|tjdS)z(Forward pass through RepNCSPELAN4 layer.rSrc3:K|]}|dVdSrrrs rMrz'RepNCSPELAN4.forward..{s/::!!AbE((::::::rN) rrlrrrorprrCrrs @rMrZzRepNCSPELAN4.forwardxsv !""1a(( ) ) ::::dh%9::::::xx !Q(((rNc2t|||j|jfdfd|j|jfD|tj dS)rrc3:K|]}|dVdSrrrs rMrz-RepNCSPELAN4.forward_split..s/88a1R5888888rN) rrlrrrrorprrCrrs @rMrzRepNCSPELAN4.forward_split~s !""DFDF#3Q77 8 8 8888DHdh#7888888xx !Q(((rNr) r6r7rgr7r r7rr7rr7r[rrbs@rMr/r/dsoO1111111")))) ))))))))rNr/c$eZdZdZdfd ZxZS) rz!ELAN1 module with 4 convolutions.r6r7rgr rc.t|||||dz|_t||dd|_t|dz|dd|_t||dd|_t|d|zz|dd|_dS)zInitialize ELAN1 layer. Args: c1 (int): Input channels. c2 (int): Output channels. c3 (int): Intermediate channels. c4 (int): Intermediate channels for convolutions. rSrriN)r=r>rrrlrorpr)rJr6rgr rrLs rMr>zELAN1.__init__s RR(((qB1%%aQ**B1%%a"f r1a00rN)r6r7rgr7r r7rr7rrbs@rMrrsC++1111111111rNrc,eZdZdZd fd Zd d ZxZS) rzAConv.r6r7rgcxtt||ddd|_dS)z}Initialize AConv module. Args: c1 (int): Input channels. c2 (int): Output channels. rirSrN)r=r>rrlrJr6rgrLs rMr>zAConv.__init__s6 B1a((rNrKrOrPc~tjj|ddddd}||S)z!Forward pass through AConv layer.rSrrFT)rCr? functional avg_pool2drlrrs rMrZz AConv.forwards4 H  * *1aAud C Cxx{{rNr6r7rgr7r[r\rbs@rMrrsVL))))))rNrc,eZdZdZd fd Zd d ZxZS) rzADown.r6r7rgct|dz|_t|dz|jddd|_t|dz|jddd|_dS)z}Initialize ADown module. Args: c1 (int): Input channels. c2 (int): Output channels. rSrirrN)r=r>rrrlrors rMr>zADown.__init__sd qaAq11aAq11rNrKrOrPcVtjj|ddddd}|dd\}}||}tjj|ddd}||}tj||fdS)z!Forward pass through ADown layer.rSrrFTri) rCr?rrrrl max_pool2dror)rJrKrrs rMrZz ADown.forwards H  * *1aAud C CAB XXb\\ X + +B1a 8 8 XXb\\y"b1%%%rNrr[r\rbs@rMrrsVL 2 2 2 2 2 2&&&&&&&&rNrc.eZdZdZd dfd Zdd ZxZS)rz SPP-ELAN.rr6r7rgr rkcrt||_t||dd|_t j|d|dz|_t j|d|dz|_t j|d|dz|_ td|z|dd|_ dS)zInitialize SPP-ELAN block. Args: c1 (int): Input channels. c2 (int): Output channels. c3 (int): Intermediate channels. k (int): Kernel size for max pooling. rrSrrRN) r=r>rrrlr?rrorprcv5)rJr6rgr rkrLs rMr>zSPPELAN.__init__s B1%%.s/BBa1R5BBBBBBrNr)rlrrorprrrCrrs @rMrZzSPPELAN.forwardsd XXa[[M BBBBDHdh#ABBBBBBxx !Q(((rN)r)r6r7rgr7r r7rkr7r[r\rbs@rMrrs[O*******"))))))))rNrc.eZdZdZddfd ZddZxZS)r'z CBLinear.rNr6r7c2s list[int]rkrr int | Nonerc t||_tj|t |||t |||d|_dS)aInitialize CBLinear module. Args: c1 (int): Input channels. c2s (list[int]): List of output channel sizes. k (int): Kernel size. s (int): Stride. p (int | None): Padding. g (int): Groups. T)groupsr:N)r=r>rr?r@sumr rB)rJr6rrkrrrrLs rMr>zCBLinear.__init__sU Ib#c((Aq'!Q--PTUUU rNrKrOrPrcc`|||jdS)z$Forward pass through CBLinear layer.rr)rBrrrrs rMrZzCBLinear.forwards'yy||!!$(!222rN)rrNr) r6r7rrrkr7rr7rrrr7)rKrOrPrcr\rbs@rMr'r'sbO V V V V V V V33333333rNr'c,eZdZdZd fd Zd d ZxZS) r&zCBFuse.idxrcVt||_dS)zmInitialize CBFuse module. Args: idx (list[int]): Indices for feature selection. N)r=r>r)rJrrLs rMr>zCBFuse.__init__s& rNxsrcrPrOc|djddfdt|ddD}tjtj||ddzdS)zForward pass through CBFuse layer. Args: xs (list[torch.Tensor]): List of input tensors. Returns: (torch.Tensor): Fused output tensor. rrSNcfg|]-\}}tj|j|d.S)nearestsizemode)r interpolater)rrrKrJ target_sizes rMrz"CBFuse.forward.. s;nnnSWSTVWq}Qtx{^+INNNnnnrNrr)rT enumeraterCrstack)rJrresrs` @rMrZzCBFuse.forwardswfl122& nnnnn[deghkikhkel[m[mnnnyS2bcc7]33;;;;rN)rr)rrcrPrOr\rbs@rMr&r&sVM < < < < < < < .$s;ll^_z"b(AAQUXYYYllllllrNN) r=r>r7rrlrorpr?rrrrs `` @rMr>z C3f.__init__s  a[[B1%%B1%%Q" b!,,llllllchijckcklllllrNrKrOrPc||||gfd|jD|t jdS)zForward pass through C3f layer.c3:K|]}|dVdSrrrs rMrzC3f.forward..)rrNr)rorlrrrprCrrs @rMrZz C3f.forward&sf XXa[[$((1++ & ****46******xx !Q(((rNrrr[r\rbs@rMrrscFFmmmmmmm$))))))))rNrc2eZdZdZ ddfd ZxZS)r$rrFrTr6r7rgrc3krrrErHrrc t||||tjfdt |D_dS)aInitialize C3k2 module. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of blocks. c3k (bool): Whether to use C3k blocks. e (float): Expansion ratio. attn (bool): Whether to use attention blocks. g (int): Groups for convolutions. shortcut (bool): Whether to use shortcut connections. c 3FK|]}rZtjtjjt jdt jdzdn:rt jjdntjjVdS)r@r attn_ratio num_headsrSN)r?rrrPSABlockr<C3k)rrXrHrrrJrs rMrz C3k2.__init__..Hs      9BM46468Q77C3tv|Q;O;OPPP    9TVTVQ!444DFDFHa88       rNNr=r>r?rrr) rJr6rgrrrrHrrrLs ` ` ```rMr>z C3k2.__init__0s. RHa333         1XX      rN)rFrFrT)r6r7rgr7rr7rrrrErHrrr7rrrrbs@rMr$r$-s\FF " " " " " " " " " " " rNr$c&eZdZdZddfd ZxZS)rzhC3k is a CSP bottleneck module with customizable kernel sizes for feature extraction in neural networks.rTrrir6r7rgrrrrrrErkct||||t||ztjfdt |D|_dS)agInitialize C3k module. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of Bottleneck blocks. shortcut (bool): Whether to use shortcut connections. g (int): Groups for convolutions. e (float): Expansion ratio. k (int): Kernel size. c 3FK|]}tfdVdS)rrNr)rrXrfrrkrs rMrzC3k.__init__..gs= d dVWBHaAq6S!Q!Q!Q d d d d d drNNr) rJr6rgrrrrrkrfrLs `` `@rMr>z C3k.__init__Xst RHa333 a[[ d d d d d d d[`ab[c[c d d derN)rTrrri)r6r7rgr7rr7rrrr7rrErkr7rrbs@rMrrUsSrrfffffffffffrNrcdeZdZdZd fd Zd d Zd d Zejd Z xZ S)r0z\RepVGGDW is a class that represents a depth-wise convolutional block in RepVGG architecture.edr7rPNonec tt||ddd|d|_t||ddd|d|_||_t j|_dS)zdInitialize RepVGGDW module. Args: ed (int): Input and output channels. r&rriFrrwN) r=r>rrBconv1rr?rrw)rJrrLs rMr>zRepVGGDW.__init__mso RAqBE::: "b!QRU;;; 799rNrKrOc~|||||zS)zPerform a forward pass of the RepVGGDW block. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after applying the depth-wise convolution. )rwrBrrrs rMrZzRepVGGDW.forwardys/xx ! tzz!}}4555rNcR|||S)zPerform a forward pass of the fused RepVGGDW block. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after applying the depth-wise convolution. )rwrBrrs rMr~zRepVGGDW.forward_fuses xx ! %%%rNct|dsdSt|jj|jj}t|jj|jj}|j}|j}|j}|j}tjj |gd}||z}||z}|jj ||jj |||_|`dS)zFuse the convolutional layers in the RepVGGDW block. This method fuses the convolutional layers and updates the weights and biases accordingly. rN)rSrSrSrS) hasattrrrBrrrHr:rCr?rrrIcopy_) rJrBrconv_wconv_bconv1_wconv1_b final_conv_w final_conv_bs rMrz RepVGGDW.fuses tW%%  F  == $*-@@,*(%))'<<<@@' '  |,,, \*** JJJrN)rr7rPrr[) r]r^r_r`r>rZr~rCno_gradrrarbs@rMr0r0jsff       6 6 6 6 & & & &U]___rNr0c.eZdZdZddfd ZddZxZS)raCompact Inverted Block (CIB) module. Args: c1 (int): Number of input channels. c2 (int): Number of output channels. shortcut (bool, optional): Whether to add a shortcut connection. Defaults to True. e (float, optional): Scaling factor for the hidden channels. Defaults to 0.5. lk (bool, optional): Whether to use RepVGGDW for the third convolutional layer. Defaults to False. TrFr6r7rgrrrrElkc tt||z}tjt ||d|t |d|zd|rt d|znt d|zd|zdd|zt d|z|dt ||d||_|o||k|_dS)aInitialize the CIB module. Args: c1 (int): Input channels. c2 (int): Output channels. shortcut (bool): Whether to use shortcut connection. e (float): Expansion ratio. lk (bool): Whether to use RepVGGDW. rir rSrN) r=r>r7r?rrr0rlr)rJr6rgrrrrfrLs rMr>z CIB.__init__s  a[[= Rb ! ! ! QVQ   " IHQV   QVQVQ!b&(I(I(I RQ   Rb ! ! !   (brNrKrOrPcj|jr|||zn||S)zForward pass of the CIB module. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor. )rrlrrs rMrZz CIB.forwards-#'(;q488A;; ;rN)TrF) r6r7rgr7rrrrErrr[r\rbs@rMrrs`))))))), < < < < < < < .s:]]qs46468srJJJ]]]]]]rNNr) rJr6rgrrrrrrLs ` `` rMr>zC2fCIB.__init__sd RHa333]]]]]]TYZ[T\T\]]]]]rN)rFFrr)r6r7rgr7rr7rrrrrr7rrErrbs@rMr!r!s^  nq^^^^^^^^^^^rNr!c.eZdZdZddfd Zdd ZxZS)raAttention module that performs self-attention on the input tensor. Args: dim (int): The input tensor dimension. num_heads (int): The number of attention heads. attn_ratio (float): The ratio of the attention key dimension to the head dimension. Attributes: num_heads (int): The number of attention heads. head_dim (int): The dimension of each attention head. key_dim (int): The dimension of the attention key. scale (float): The scaling factor for the attention scores. qkv (Conv): Convolutional layer for computing the query, key, and value. proj (Conv): Convolutional layer for projecting the attended values. pe (Conv): Convolutional layer for positional encoding. rPrrr7rrrEcxt||_||z|_t |j|z|_|jdz|_|j|z}||dzz}t||dd|_t||dd|_ t||dd|d|_ dS) zInitialize multi-head attention module. Args: dim (int): Input dimension. num_heads (int): Number of attention heads. attn_ratio (float): Attention ratio for key dimension. rSrFrvrirN) r=r>rhead_dimr7key_dimr1rqkvr^pe)rJrrrnh_kdr@rLs rMr>zAttention.__init__ s "y( 4=:566 \4'  y( %!)OQu---c1%000 sCA%888rNrKrOrPc P|j\}}}}||z}||}|||j|jdz|jz||j|j|jgd\}} } |dd| z|jz} | d} | | ddz||||| | ||||z}| |}|S)zForward pass of the Attention module. Args: x (torch.Tensor): The input tensor. Returns: (torch.Tensor): The output tensor after self-attention. rSrr) rTrrGrrrrrUr1rVrrir^) rJrKBCHWNrrjrkrkrHs rMrZzAttention.forwardsW 1a Ehhqkk((1dndlQ.>.NPQRRXX \4< 7QY  1a B##a'4:5|||## B'' ' - -aAq 9 9DGGAIIaQRTUWXDYDY>> psablock = PSABlock(c=128, attn_ratio=0.5, num_heads=4, shortcut=True) >>> input_tensor = torch.randn(1, 128, 32, 32) >>> output_tensor = psablock(input_tensor) rrRTrr7rrErrrrPrc tt||||_t jt ||dzdt |dz|dd|_||_dS)aInitialize the PSABlock. Args: c (int): Input and output channels. attn_ratio (float): Attention ratio for key dimension. num_heads (int): Number of attention heads. shortcut (bool): Whether to use shortcut connections. rrSrFrvN) r=r>rrHr?rrffnr)rJrrrrrLs rMr>zPSABlock.__init__Hst aJ)LLL =aQ!2!2DQ1%4P4P4PQQrNrKrOc|jr|||zn||}|jr|||zn||}|S)zExecute a forward pass through PSABlock. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after attention and feed-forward processing. )rrHrrrs rMrZzPSABlock.forwardWs[!% :A !  diill#x 8A OOTXXa[[rN)rrRT) rr7rrErr7rrrPrr[r\rbs@rMrr3s`(               rNrc.eZdZdZd dfd Zdd ZxZS)raPSA class for implementing Position-Sensitive Attention in neural networks. This class encapsulates the functionality for applying position-sensitive attention and feed-forward networks to input tensors, enhancing feature extraction and processing capabilities. Attributes: c (int): Number of hidden channels after applying the initial convolution. cv1 (Conv): 1x1 convolution layer to reduce the number of input channels to 2*c. cv2 (Conv): 1x1 convolution layer to reduce the number of output channels to c1. attn (Attention): Attention module for position-sensitive attention. ffn (nn.Sequential): Feed-forward network for further processing. Methods: forward: Applies position-sensitive attention and feed-forward network to the input tensor. Examples: Create a PSA module and apply it to an input tensor >>> psa = PSA(c1=128, c2=128, e=0.5) >>> input_tensor = torch.randn(1, 128, 64, 64) >>> output_tensor = psa.forward(input_tensor) rr6r7rgrrEc  t||ksJt||z|_t |d|jzdd|_t d|jz|d|_t|jdt|jdzd|_ tj t |j|jdzdt |jdz|jdd|_ dS) zInitialize PSA module. Args: c1 (int): Input channels. c2 (int): Output channels. e (float): Expansion ratio. rSrrrrFrvN) r=r>r7rrrlrorr<rHr?rr)rJr6rgrrLs rMr>z PSA.__init__|s RxxxxR!VAJ1--DF B**dfDFbLRS@T@TUUU =dfdfqj!!d>d>deerNrKrOrPc(|||j|jfd\}}|||z}|||z}|t j||fdS)zExecute forward pass in PSA module. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after attention and feed-forward processing. rr)rlrrrHrrorCrrs rMrZz PSA.forwards|xx{{  $&$&!1q 991  !   Oxx 1a&!,,---rN)r)r6r7rgr7rrEr[r\rbs@rMrresg,fffffff" . . . . . . . .rNrc.eZdZdZddfd ZddZxZS)raHC2PSA module with attention mechanism for enhanced feature extraction and processing. This module implements a convolutional block with attention mechanisms to enhance feature extraction and processing capabilities. It includes a series of PSABlock modules for self-attention and feed-forward operations. Attributes: c (int): Number of hidden channels. cv1 (Conv): 1x1 convolution layer to reduce the number of input channels to 2*c. cv2 (Conv): 1x1 convolution layer to reduce the number of output channels to c1. m (nn.Sequential): Sequential container of PSABlock modules for attention and feed-forward operations. Methods: forward: Performs a forward pass through the C2PSA module, applying attention and feed-forward operations. Examples: >>> c2psa = C2PSA(c1=256, c2=256, n=3, e=0.5) >>> input_tensor = torch.randn(1, 256, 64, 64) >>> output_tensor = c2psa(input_tensor) Notes: This module essentially is the same as PSA module, but refactored to allow stacking more PSABlock modules. rrr6r7rgrrrEcZt||ksJt||z_t |djzdd_t djz|d_tjfdt|D_ dS)zInitialize C2PSA module. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of PSABlock modules. e (float): Expansion ratio. rSrc3VK|]#}tjdjdzV$dS)rrrN)rrrrXrJs rMrz!C2PSA.__init__..s< l l^_$&SDFVXL!Y!Y!Y l l l l l lrNNrrJr6rgrrrLs` rMr>zC2PSA.__init__s RxxxxR!VAJ1--DF B** l l l lchijckck l l lmrNrKrOrPc|||j|jfd\}}||}|t j||fdS)zProcess the input tensor through a series of PSA blocks. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after processing. rr)rlrrrrorCrrs rMrZz C2PSA.forwardsdxx{{  $&$&!1q 991 FF1IIxx 1a&!,,---rNrrrr[r\rbs@rMrrsg.nnnnnnn" . . . . . . . .rNrc&eZdZdZd d fd ZxZS) r"aC2fPSA module with enhanced feature extraction using PSA blocks. This class extends the C2f module by incorporating PSA blocks for improved attention mechanisms and feature extraction. Attributes: c (int): Number of hidden channels. cv1 (Conv): 1x1 convolution layer to reduce the number of input channels to 2*c. cv2 (Conv): 1x1 convolution layer to reduce the number of output channels to c2. m (nn.ModuleList): List of PSABlock modules for feature extraction. Methods: forward: Performs a forward pass through the C2fPSA module. forward_split: Performs a forward pass using split() instead of chunk(). Examples: >>> import torch >>> from ultralytics.nn.modules.block import C2fPSA >>> model = C2fPSA(c1=64, c2=64, n=3, e=0.5) >>> x = torch.randn(1, 64, 128, 128) >>> output = model(x) >>> print(output.shape) rrr6r7rgrrrEc||ksJt||||tjfdt |D_dS)zInitialize C2fPSA module. Args: c1 (int): Input channels. c2 (int): Output channels. n (int): Number of PSABlock modules. e (float): Expansion ratio. )rrc 3rK|]1}tjdtjdzdV2dS)rrrrN)rrr<rs rMrz"C2fPSA.__init__..sGrrdex3#dfXZl\]J^J^___rrrrrrrNNrrs` rMr>zC2fPSA.__init__shRxxxx R1***rrrrinopiqiqrrrrrrNrrrrbs@rMr"r"sW0 s s s s s s s s s s srNr"c,eZdZdZd fd Zd d ZxZS)r2aHSCDown module for downsampling with separable convolutions. This module performs downsampling using a combination of pointwise and depthwise convolutions, which helps in efficiently reducing the spatial dimensions of the input tensor while maintaining the channel information. Attributes: cv1 (Conv): Pointwise convolution layer that reduces the number of channels. cv2 (Conv): Depthwise convolution layer that performs spatial downsampling. Methods: forward: Applies the SCDown module to the input tensor. Examples: >>> import torch >>> from ultralytics.nn.modules.block import SCDown >>> model = SCDown(c1=64, c2=128, k=3, s=2) >>> x = torch.randn(1, 64, 128, 128) >>> y = model(x) >>> print(y.shape) torch.Size([1, 128, 64, 64]) r6r7rgrkrctt||dd|_t|||||d|_dS)zInitialize SCDown module. Args: c1 (int): Input channels. c2 (int): Output channels. k (int): Kernel size. s (int): Stride. rF)rkrrrwN)r=r>rrlro)rJr6rgrkrrLs rMr>zSCDown.__init__sP B1%%B!qBE:::rNrKrOrPcR|||S)zApply convolution and downsampling to the input tensor. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Downsampled output tensor. )rorlrrs rMrZzSCDown.forwards xx $$$rNrr[r\rbs@rMr2r2s[, ; ; ; ; ; ; % % % % % % % %rNr2c0eZdZdZ ddfd ZddZxZS)r3a?TorchVision module to allow loading any torchvision model. This class provides a way to load a model from the torchvision library, optionally load pre-trained weights, and customize the model by truncating or unwrapping layers. Args: model (str): Name of the torchvision model to load. weights (str, optional): Pre-trained weights to load. Default is "DEFAULT". unwrap (bool, optional): Unwraps the model to a sequential containing all but the last `truncate` layers. truncate (int, optional): Number of layers to truncate from the end if `unwrap` is True. Default is 2. split (bool, optional): Returns output from intermediate child modules as list. Default is False. Attributes: m (nn.Module): The loaded torchvision model, possibly truncated and unwrapped. DEFAULTTrSFmodelstrweightsunwraprtruncater7rcddl}tt|jdr"|j|||_n.|jj|t||_|rt|j }t|dtj r3gt|d |dd}tj |r |d| n||_||_dSd|_tjx|j_|j_dS)aeLoad the model and weights from torchvision. Args: model (str): Name of the torchvision model to load. weights (str): Pre-trained weights to load. unwrap (bool): Whether to unwrap the model. truncate (int): Number of layers to truncate. split (bool): Whether to split the output. rN get_model)r) pretrainedrF) torchvisionr=r>rmodelsrr__dict__rrchildren isinstancer?rrrheadheads) rJr rrrrrlayersrLs rMr>zTorchVision.__init__;s;   ;%{ 3 3 R '11%1IIDFF7['074==QQQDF  7$&//++,,F&)R]33 DC4q 2 2 4 455Cqrr C]8%OVJhYJ%7%7QDFDJJJDJ)+ 6DFK$&,,,rNrKrOrPc|jr*|gfd|jDn||S)zForward pass through the model. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor | list[torch.Tensor]): Output tensor or list of tensors. c3:K|]}|dVdSrrrs rMrz&TorchVision.forward..cs/..!QQquXX......rN)rrrrs @rMrZzTorchVision.forwardXsV : A HH....tv... . . . .q ArN)r TrSF) r rrrrrrr7rrr[r\rbs@rMr3r3*sg"kp7777777:rNr3c.eZdZdZd d fd Zdd ZxZS)AAttnaArea-attention module for YOLO models, providing efficient attention mechanisms. This module implements an area-based attention mechanism that processes input features in a spatially-aware manner, making it particularly effective for object detection tasks. Attributes: area (int): Number of areas the feature map is divided into. num_heads (int): Number of heads into which the attention mechanism is divided. head_dim (int): Dimension of each attention head. qkv (Conv): Convolution layer for computing query, key and value tensors. proj (Conv): Projection convolution layer. pe (Conv): Position encoding convolution layer. Methods: forward: Applies area-attention to input tensor. Examples: >>> attn = AAttn(dim=256, num_heads=8, area=4) >>> x = torch.randn(1, 256, 32, 32) >>> output = attn(x) >>> print(output.shape) torch.Size([1, 256, 32, 32]) rrr7rareac ,t||_||_||zx|_}||jz}t ||dzdd|_t ||dd|_t ||ddd|d|_dS)a#Initialize an Area-attention module for YOLO models. Args: dim (int): Number of hidden channels. num_heads (int): Number of heads into which the attention mechanism is divided. area (int): Number of areas the feature map is divided into. rirFrvr&rN) r=r>r!rrrrr^r)rJrrr!r all_head_dimrLs rMr>zAAttn.__init__s  "#&)#33 $.0 \A-qe<<<sA5999 |S!QSeDDDrNrKrOrPc|j\}}}}||z}||ddd}|jdkr5|||jz||jz|dz}|j\}}}||||j|jdz dddd |j|j|jgd\} } } | dd| z|jdzz} | d} | | ddz}| dddd}| dddd} |jdkrY|||jz||jz|}| ||jz||jz|} |j\}}}||||| dddd }| |||| dddd } || | z}||S) zProcess the input tensor through the area-attention. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after area-attention. rSrrirrrrr)rTrflattenrUr!rirGrrpermuterrV contiguousrr^) rJrKrrrrrrrXrjrkrkrHs rMrZz AAttn.forwardsJW 1a Ehhqkk!!!$$..q!44 9q==++a$)mQ$)^QUCCCiGAq! HHQ4>4=1+< = = WQ1a UDM4=$-@aU H H 1a  B##a'DM4,?@|||## r2&& & IIaAq ! ! IIaAq ! ! 9q== !ty.!di-;;A !ty.!di-;;AgGAq! IIaAq ! ! ) )!Q1 5 5 @ @ B B IIaAq ! ! ) )!Q1 5 5 @ @ B B  Nyy||rNr)rr7rr7r!r7r[r\rbs@rMr r isg0EEEEEEE&$$$$$$$$rNr cFeZdZdZddfd Zedd ZddZxZS)ABlockaArea-attention block module for efficient feature extraction in YOLO models. This module implements an area-attention mechanism combined with a feed-forward network for processing feature maps. It uses a novel area-based attention approach that is more efficient than traditional self-attention while maintaining effectiveness. Attributes: attn (AAttn): Area-attention module for processing spatial features. mlp (nn.Sequential): Multi-layer perceptron for feature transformation. Methods: _init_weights: Initializes module weights using truncated normal distribution. forward: Applies area-attention and feed-forward processing to input tensor. Examples: >>> block = ABlock(dim=256, num_heads=8, mlp_ratio=1.2, area=1) >>> x = torch.randn(1, 256, 32, 32) >>> output = block(x) >>> print(output.shape) torch.Size([1, 256, 32, 32]) 333333?rrr7r mlp_ratiorEr!c Btt||||_t ||z}t jt||dt||dd|_| |j dS)aaInitialize an Area-attention block module. Args: dim (int): Number of input channels. num_heads (int): Number of heads into which the attention mechanism is divided. mlp_ratio (float): Expansion ratio for MLP hidden dimension. area (int): Number of areas the feature map is divided into. )rr!rFrvN) r=r>r rHr7r?rrmlpapply _init_weights)rJrrr+r!mlp_hidden_dimrLs rMr>zABlock.__init__s #>>> S9_--=c>1!=!=tNTWYZ`e?f?f?fgg 4%&&&&&rNrrct|tjrTtj|jd|j)tj|jddSdSdS)zInitialize weights using a truncated normal distribution. Args: m (nn.Module): Module to initialize. g{Gz?)stdNr)rr?r@init trunc_normal_rHr: constant_rs rMr/zABlock._init_weightssm a # # - G ! !!( ! 5 5 5v!!!!&!,,,,, - -!!rNrKrOrPcb|||z}|||zS)zForward pass through ABlock. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after area-attention and feed-forward processing. )rHr-rrs rMrZzABlock.forwards,  ! 488A;;rN)r*r)rr7rr7r+rEr!r7)rrr[) r]r^r_r`r>rr/rZrarbs@rMr)r)s,'''''''" - - -\ -        rNr)c>eZdZdZ ddfd ZddZxZS)A2C2faArea-Attention C2f module for enhanced feature extraction with area-based attention mechanisms. This module extends the C2f architecture by incorporating area-attention and ABlock layers for improved feature processing. It supports both area-attention and standard convolution modes. Attributes: cv1 (Conv): Initial 1x1 convolution layer that reduces input channels to hidden channels. cv2 (Conv): Final 1x1 convolution layer that processes concatenated features. gamma (nn.Parameter | None): Learnable parameter for residual scaling when using area attention. m (nn.ModuleList): List of either ABlock or C3k modules for feature processing. Methods: forward: Processes input through area-attention or standard convolution pathway. Examples: >>> m = A2C2f(512, 512, n=1, a2=True, area=1) >>> x = torch.randn(1, 512, 32, 32) >>> output = m(x) >>> print(output.shape) torch.Size([1, 512, 32, 32]) rTF@rr6r7rgra2rr!residualr+rErrrc  tt||z dzdks Jdt| dd|_td|z z|d|_r-|r+t jdtj |zdnd|_ t j   fd t|D|_ dS) aInitialize Area-Attention C2f module. Args: c1 (int): Number of input channels. c2 (int): Number of output channels. n (int): Number of ABlock or C3k modules to stack. a2 (bool): Whether to use area attention blocks. If False, uses C3k blocks instead. area (int): Number of areas the feature map is divided into. residual (bool): Whether to use residual connections with learnable gamma parameter. mlp_ratio (float): Expansion ratio for MLP hidden dimension. e (float): Channel expansion ratio for hidden channels. g (int): Number of groups for grouped convolutions. shortcut (bool): Whether to use shortcut connections in C3k blocks. rerz-Dimension of ABlock must be a multiple of 32.r{Gz?TrTNc3K|]B}r)tjfdtdDntdVCdS)c3BK|]}tdzVdS)reN)r))rrXr!rfr+s rMrz+A2C2f.__init__...8s5TTaF2rRxDAATTTTTTrNrSN)r?rrr)rrXr:r!rfrr+rs rMrz!A2C2f.__init__..7s   -BMTTTTTT5QR88TTT U URQ!,,      rN)r=r>r7rrlror?rFrCr8gammarrr) rJr6rgrr:r!r;r+rrrrfrLs `` ` ``@rMr>zA2C2f.__init__s 6  a[[Bw!|||L|||B1%%Q" b!,,PRiW_iR\$B"7tLLLLei          1XX      rNrKrOrPcD||gfd|jD|t jd|j3||jd|jjdddzzSS)zForward pass through A2C2f layer. Args: x (torch.Tensor): Input tensor. Returns: (torch.Tensor): Output tensor after processing. c3:K|]}|dVdSrrrs rMrz A2C2f.forward..HrrNrNrr) rlrrrorCrr@rGrTrs @rMrZz A2C2f.forward>sXXa[[M ****46****** HHUYq!__ % % : !tzr4:+;A+>1EEII IrN)rTrFr9rrT)r6r7rgr7rr7r:rr!r7r;rr+rErrErr7rrr[r\rbs@rMr8r8s{4( ( ( ( ( ( ( TrNr8c.eZdZdZd dfd Zdd ZxZS) SwiGLUFFNz@SwiGLU Feed-Forward Network for transformer-based architectures.rRr0r7rrrPrcttj|||z|_tj||zdz||_dS)zInitialize SwiGLU FFN with input dimension, output dimension, and expansion factor. Args: gc (int): Guide channels. ec (int): Embedding channels. e (int): Expansion factor. rSN)r=r>r?r4w12w3)rJr0rrrLs rMr>zSwiGLUFFN.__init__RsQ 9RR(()AFaK,,rNrKrOc||}|dd\}}tj||z}||S)z.Apply SwiGLU transformation to input features.rSrr)rFrrsilurG)rJrKx12rrhiddens rMrZzSwiGLUFFN.forward^sKhhqkk1"%%BbwwvrN)rR)r0r7rr7rr7rPrr[r\rbs@rMrDrDOs\JJ - - - - - - -rNrDc,eZdZdZd fd Zd d ZxZS) Residualz7Residual connection wrapper for neural network modules.rrrPrct||_tj|jjjtj|jjjdS)zInitialize residual module with the wrapped module. Args: m (nn.Module): Module to wrap with residual connection. N) r=r>rr?r3zeros_rGr:rH)rJrrLs rMr>zResidual.__init__is_  tvy~&&& tvy'(((((rNrKrOc2|||zS)z,Apply residual connection to input features.rrrs rMrZzResidual.forwardvs466!99}rN)rrrPrr[r\rbs@rMrMrMfsWAA ) ) ) ) ) )rNrMc,eZdZdZdfd Zdd ZxZS)SAVPEzESpatial-Aware Visual Prompt Embedding module for feature enhancement.rQrr r7rBc ttjfdt |D|_tjfdt |D|_d|_tjdz|d|_ tjdz|jdd|_ tjd|jdd|_ tj td|jz|jdtj|j|jdd|_dS) aInitialize SAVPE module with channels, intermediate channels, and embedding dimension. Args: ch (list[int]): List of input channel dimensions. c3 (int): Intermediate channels. embed (int): Embedding dimension. c 3K|]h\}}tjt|dtd|dvrtj|dzntjVidS)rirrSrS scale_factorNr?rrUpsamplerrrrKr s rMrz!SAVPE.__init__..s! ! 1 MQARQTUY_T_T_!a%1P1P1P1Pegeperer  ! ! ! ! ! ! rNc3K|]X\}}tjt|d|dvrtj|dzntjVYdS)rrUrSrVNrXrZs rMrz!SAVPE.__init__..sz! ! 1 M$q"a..QRX[["+1q5*I*I*I*I^`^i^k^k l l! ! ! ! ! ! rNr5rir)rzrSN)r=r>r?rrrlrorr@rprrrrcv6)rJrQr rBrLs ` rMr>zSAVPE.__init__~sB =! ! ! ! "" ! ! !   =! ! ! ! !" ! ! !    9QVUA..9QVTVQ:::9Q1555=a$&j$&!!e>e>effrNrKrcvprOrPcRfdt|D}tj|d}fdt|D}tj|d}|j\}}}}|jd}|||d}||dj|| d|ddd||zj||}|||d||||zd||} tj| |fd}|||jd}|||dd}||ztj |tj |jjzz} t!j| d|j} | dd||j|jzdddz} t!j| dd||ddd S) zJProcess input features and visual prompts to generate enhanced embeddings.cBg|]\}}j||Sr)rorrxirJs rMrz!SAVPE.forward..+ 7 7 7B[TXa[__ 7 7 7rNrrcBg|]\}}j||Sr)rlr`s rMrz!SAVPE.forward..rbrNrrrSrs)rrrCrrprTrGrirexpandr\r logical_notfinfor<minrrVtorUru) rJrKr]rrrrrQscore aggregateds ` rMrZz SAVPE.forwards` 7 7 7 7)A,, 7 7 7 HHUYqa((( ) ) 7 7 7 7)A,, 7 7 7 HHUYqa((( ) )W 1a HQK FF1a   IIaDFAq ) ) 0 0QB C C K KAPQESWSY[\^_ ` ` ZZ1aA & & . .q1uaA > > HHUY488B<<0a888 9 9 IIaDFB ' ' ZZ1a $ $B*2..QW1E1E1III %R(((++AG44__R,,qyyDFAKQS/T/T/^/^_ace/f/ff {://B77??1bIIrUVWWWWrN)rQrr r7rBr7)rKrcr]rOrPrOr\rbs@rMrRrR{seOOgggggg6XXXXXXXXrNrRc:eZdZdZddfd Zddfd ZdZxZS)Proto26zDUltralytics YOLO26 models mask Proto module for segmentation models.rrdrePrQtuplerfr7rgncc t|||tjfdddD|_t d|d|_tjt d|dt ||dtj||d|_ dS)apInitialize the Ultralytics YOLO models mask Proto module with specified number of protos and masks. Args: ch (tuple): Tuple of channel sizes from backbone feature maps. c_ (int): Intermediate channels. c2 (int): Output channels (number of protos). nc (int): Number of classes for semantic segmentation. c3HK|]}t|ddVdS)rrrjNr)rrKrQs rMrz#Proto26.__init__..s6(M(MaA!)<)<)<(M(M(M(M(M(MrNrNrrirj) r=r>r?r feat_refiner feat_fuserr@semseg)rJrQrfrgrqrLs ` rMr>zProto26.__init__s R$$$=(M(M(M(Mbf(M(M(MMMbeR1---mDAa$8$8$8$r2:K:K:KRYWY[]_`MaMabb rNTrKrO return_semsegrrPcr|d}t|jD]B\}}|||dz}tj||jddd}||z}Ct ||}|jr|r| |}||fS|S)zUPerform a forward pass by fusing multi-scale feature maps and generating proto masks.rrrSNrr) rrtrrrTr=rZrutrainingrv) rJrKrwfeatrfup_featrrvrLs rMrZzProto26.forwardstd.// " "DAqa!a%kkGmG$*QRR.yQQQG'>DD GGOODNN400 1 1 = ] [[&&Fv; rNcd|_dS)zHFuse the model for inference by removing the semantic segmentation head.N)rvrs rMrz Proto26.fuses  rN)rrdrero)rQrprfr7rgr7rqr7)T)rKrOrwrrPrO)r]r^r_r`r>rZrrarbs@rMrnrnsNN c c c c c c c       rNrncveZdZdZedZedZedZfdZ dZ dZ dZ xZ S) RealNVPzRealNVP: a flow-based generative model. References: https://arxiv.org/abs/1605.08803 https://github.com/open-mmlab/mmpose/blob/main/mmpose/models/utils/realnvp.py c  tjtjddtjtjddtjtjddtjS)z3Get the scale model in a single invertible mapping.rSr)r?rr4rTanhrrNrMnetsz RealNVP.netss\}RYq"--rwyy")B:K:KRWYYXZXabdfgXhXhjljqjsjstttrNc tjtjddtjtjddtjtjddS)z9Get the translation model in a single invertible mapping.rSr)r?rr4rrrNrMnettz RealNVP.nettsP}RYq"--rwyy")B:K:KRWYYXZXabdfgXhXhiiirNcVtj|j|jS)zThe prior distribution.)rC distributionsMultivariateNormalloccovrs rMpriorz RealNVP.priors!"55dhIIIrNctdtjddtjddtjddgddggdztjtj fd ttj D_ tj fd ttj D_dS) NrrSrmaskrrrir;c8g|]}Sr)rrs rMrz$RealNVP.__init__..!%Q%Q%Qadiikk%Q%Q%QrNc8g|]}Sr)rrs rMrz$RealNVP.__init__..rrN)r=r>register_bufferrCr6eyer_float32r?rrrrrt init_weightsrqs`rMr>zRealNVP.__init__s  UEKNN333 UEIaLL111 VU\Aq6Aq62BQ2Fem%\%\%\]]]$$%Q%Q%Q%Q5TY;P;P%Q%Q%QRR$$%Q%Q%Q%Q5TY;P;P%Q%Q%QRR rNc|D]B}t|tjr&tj|jdCdS)zInitialize model weights.r=)gainN)modulesrr?r4r3xavier_uniform_rH)rJrs rMrzRealNVP.init_weightssU = =A!RY'' =''t'<<< = =rNc||jd|}}ttt |jD]}|j||z}|j||d|j|z z}|j||d|j|z z}d|j|z ||z ztj | z|z}|| dz}||fS)zApply mapping from the data space to the latent space and calculate the log determinant of the Jacobian matrix. rrr) new_zerosrTreversedrrrrrrCrvr)rJrK log_det_jacobzrz_rrs rM backward_pzRealNVP.backward_ps;;qwqz22Aq %DF ,,-- * *A1!Bq " TYq\!12Aq " TYq\!12ATYq\!a!e,uy!}}rrrrarbs@rMrrsuu\ujj\jJJXJ     ===    0000000rNr)Lr` __future__rrCtorch.nnr?torch.nn.functionalrrultralytics.utils.torch_utilsrrBrrrr r r transformerr __all__Modulerr-r+r*rrr rrrr%r.rr#r)rrrr1r,r r,r(rrrr/rrrrr'r&rr$rr0rr!rrrrr"r2r3r r)r8rDrMrRrnrrrNrMrs} """""" ::::::FFFFFFFFFFFFFFFF))))))( V\\\\\")\\\2>>>>>BI>>>,!!!!!RY!!!H)()()()()(bi)()()(XDDDDD")DDD.;;;;;29;;;@*666666666)))))"))))DJJJJJJJJ4xxxxx"xxx&;;;;;BI;;;0000002000&MMMMMbMMM&/////bi///8PPPPPPPP6CCCCCBICCC>JJJJJ")JJJ0")>1$1$1$1$1$")1$1$1$h?)?)?)?)?)bi?)?)?)D>%>%>%>%>%ry>%>%>%B66666bi6664.6.6.6.6.6 .6.6.6b,,,,,J,,,*_____R___&)))))29)))B11111L111(BI&&&&&&BI&&&2)))))bi)))633333ry333.<<<<>>>>ux>>>B*<*<*<*<*<")*<*<*99999 999x/////ry///d4.4.4.4.4.")4.4.4.n4.4.4.4.4.BI4.4.4.n$s$s$s$s$sS$s$s$sN-%-%-%-%-%RY-%-%-%`<<<<<")<<<~PPPPPBIPPPf>>>>>RY>>>BOOOOOBIOOOd .ry*8X8X8X8X8XBI8X8X8Xv     e   F:0:0:0:0:0bi:0:0:0:0:0rN