/jv"ddlmZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZd d lmZGd d eZdS) ) annotationsN)Image)SegmentationPredictor) DEFAULT_CFG)box_iou) scale_masks) TORCH_1_10)adjust_bboxes_to_image_bordercJeZdZdZeddfd fd ZfdZd dZdZd Z xZ S) FastSAMPredictoraFastSAMPredictor is specialized for fast SAM (Segment Anything Model) segmentation prediction tasks. This class extends the SegmentationPredictor, customizing the prediction pipeline specifically for fast SAM. It adjusts post-processing steps to incorporate mask prediction and non-maximum suppression while optimizing for single-class segmentation. Attributes: prompts (dict): Dictionary containing prompt information for segmentation (bboxes, points, labels, texts). device (torch.device): Device on which model and tensors are processed. clip (Any, optional): CLIP model used for text-based prompting, loaded on demand. Methods: postprocess: Apply postprocessing to FastSAM predictions and handle prompts. prompt: Perform image segmentation inference based on various prompt types. set_prompts: Set prompts to be used during inference. N _callbacks dict | Nonec\t|||i|_dS)a;Initialize the FastSAMPredictor with configuration and callbacks. This initializes a predictor specialized for Fast SAM (Segment Anything Model) segmentation tasks. The predictor extends SegmentationPredictor with custom post-processing for mask prediction and non-maximum suppression optimized for single-class segmentation. Args: cfg (dict): Configuration for the predictor. overrides (dict, optional): Configuration overrides. _callbacks (dict, optional): Dictionary of callback functions. N)super__init__prompts)selfcfg overridesr __class__s g/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/ultralytics/models/fastsam/predict.pyrzFastSAMPredictor.__init__#s, i444 c|jdd}|jdd}|jdd}|jdd}t|||}|D]} t jdd| jd| jdg| jjj tj } t| jj | j} t j t| d| d k} | dkr| | jj | <|||||| S) aApply postprocessing to FastSAM predictions and handle prompts. Args: preds (list[torch.Tensor]): Raw predictions from the model. img (torch.Tensor): Input image tensor that was fed to the model. orig_imgs (list[np.ndarray]): Original images before preprocessing. Returns: (list[Results]): Processed results with prompts applied. bboxesNpointslabelstextsrr )devicedtypeg?)rrrr)rpopr postprocesstorchtensor orig_shapeboxesdatarfloat32r xyxynonzerorflattennumelprompt)rpredsimg orig_imgsrrrrresultsresultfull_boxr&idxrs rr"zFastSAMPredictor.postprocess2sT!!(D11!!(D11!!(D11   $//''%%eS)<< 2 2F|Av(+V->q-AB6 > DEEMMOOCyy{{a)1 !#&{{76&W\{]]]rc  ||||Sg}t|ts|g}|D]F}t|dkr||,|jjjdd|jkrItd |jddk tj t|tj |j}|tj|tj|j}|jdkr|dn|}|dddf|dddfz |dddf|dddfz z} tjfd|D} tjd } | dddf| z| z } d |tj| | z d <|tj|tj|j}|jdkr|dn|}|tj|jd}tj|tj|j}t|t|ks+Jd t|d t|d|dkr3tjt|tj |jn2tj t|tj |j} t-||D]H\}}t|| tjdd|d|dfd d<I|| z}|t|t0r|g}ggc}t3|jjD]\}}d|D\}}}}t:r|n,|ddkr|~|j||||f|||||df z}|tCj"|dddddddf|#||}tj|d }trVfdtIt|D}tj%|tM||j}d ||<|||H|S)a Perform image segmentation inference based on cues like bounding boxes, points, and text prompts. Args: results (Results | list[Results]): Original inference results from FastSAM models without any prompts. bboxes (np.ndarray | list, optional): Bounding boxes with shape (N, 4), in XYXY format. points (np.ndarray | list, optional): Points indicating object locations with shape (N, 2), in pixels. labels (np.ndarray | list, optional): Labels for point prompts, shape (N, ). 1 = foreground, 0 = background. texts (str | list[str], optional): Textual prompts, a list containing string objects. Returns: (list[Results]): Output results filtered and determined by the provided prompts. Nrr g?)r rcg|]?}dd|d|d|d|dfd@S)Nr r6rr7r r7dim)sum).0bmaskss r z+FastSAMPredictor.prompt..lsT)m)m)mab%1Q4!A$;!qt 0K*L*P*PU[*P*\*\)m)m)mrr9r:Tz?Expected `labels` to have the same length as `points`, but got z and .)as_tuplec34K|]}t|VdS)N)int)r=xs r z*FastSAMPredictor.prompt..s(%8%8c!ff%8%8%8%8%8%8rdcg|]}|v| SrJ)r=i filter_idxs rr@z+FastSAMPredictor.prompt..s#UUUa*ATATATATATrr)' isinstancelistlenappendr?r'shaper%rfloatbyter#zerosboolr as_tensorint32ndimstackr<argmaxoneszipr*str enumerater&r)tolistr orig_imgcpunumpyr fromarray_clip_inferenceranger$rD)rr1rrrrprompt_resultsr2r4 bbox_areas mask_areasfull_mask_areasunion point_idxpointlabelcrop_imsrKr>x1y1x2y2crop similaritytext_idxori_idxsrLr?s @@rr-zFastSAMPredictor.promptMsW >fnN'4(( iG6 /6 /F6{{a%%f---L%E{122&"333$U4[%6%6%8%8&:KLLQORUU[[]]+c&kkDKPPPC!u{4;WWW)/)9)9v$QQQTlVAAAqD\9fQQQTlVTUTUTUWXTX\>YZ "[)m)m)m)mfl)m)m)mnn "')Ev">">">"111d7+o= J?CELe!3;;;<!u{4;WWW)/)9)9v>"Z Q88Fu{4;WWW6{{c&kk111vVYZ`VaVavvhklrhshsvvv211 zz||q((Js6{{%*T[QQQQS[[ 4;WWW %($7$7kkLE5_cdi_j_jIemE!!!U1XuQx2G,HSWXXXYZ[\\y  eS))$"GE')2$*%fl&7&>&>&@&@AAGGDAq%8%8a%8%8%8NBB*4Oa %(,,q//:M:M:O:OTWWW"))!,,, !?2b5"R%<85BrE2b5RVAV;W;[;[;];];c;c;e;eeDOOEODAAAttt4D$E$EFFFF!11(EBB  < ;;;z??YUUUU5V+=+=UUUH$|HS]],CDKXXXH $H  ! !&+ . . . .rcJddlm}tds|dj_t jfd|D}j|}jj |}||j zS)aPerform CLIP inference to calculate similarity between images and text prompts. Args: images (list[PIL.Image]): List of source images, each should be PIL.Image with RGB channel order. texts (list[str]): List of prompt texts, each should be a string object. Returns: (torch.Tensor): Similarity matrix between given images and texts with shape (M, N). r)CLIPclipzViT-B/32rMctg|]4}j|j5SrJ)rzimage_preprocesstor)r=imagers rr@z4FastSAMPredictor._clip_inference..s:dddTYdi88??BB4;OOdddr) ultralytics.nn.text_modelryhasattrrrzr#rZ encode_image encode_texttokenizeT)rimagesrryimage_features text_featuress` rrez FastSAMPredictor._clip_inferences 322222tV$$ =Z <<rs#""""" AAAAAA))))))------------444444000000]]]]],]]]]]r