j%ddlmZddlZddlmZddlmZddlZddl m Z ddl m Z ddl mZmZddlmZdd lmZmZd ejd <Gd d ZGddZdS)) annotationsN)Path)Any)Image) IMG_FORMATS)LOGGER TORCH_VERSION)check_requirements) TORCH_2_4 select_deviceTRUEKMP_DUPLICATE_LIB_OKcDeZdZdZddZdd Zdd ZddZdddZd dZ dS)!VisualAISearchaA semantic image search system that leverages OpenCLIP for generating high-quality image and text embeddings and FAISS for fast similarity-based retrieval. This class aligns image and text embeddings in a shared semantic space, enabling users to search large collections of images using natural language queries with high accuracy and speed. Attributes: data (str): Directory containing images. device (str): Computation device, e.g., 'cpu' or 'cuda'. faiss_index (str): Path to the FAISS index file. data_path_npy (str): Path to the numpy file storing image paths. data_dir (Path): Path object for the data directory. model: Loaded CLIP model. index: FAISS index for similarity search. image_paths (list[str]): List of image file paths. Methods: extract_image_feature: Extract CLIP embedding from an image. extract_text_feature: Extract CLIP embedding from text. load_or_build_index: Load existing FAISS index or build new one. search: Perform semantic search for similar images. Examples: Initialize and search for images >>> searcher = VisualAISearch(data="path/to/images", device="cuda") >>> results = searcher.search("a cat sitting on a chair", k=10) kwargsrreturnNonec tsJdtdddlm}t dt d|_d|_d|_t| d d |_ t| d d |_ |j sPdd lm}t#j|j d|dddlm}||dddtd |_ |d|j |_d|_g|_|dS)zDInitialize the VisualAISearch class with FAISS index and CLIP model.z1VisualAISearch requires torch>=2.4 (found torch==)r)build_text_modelz faiss-cpufaissz faiss.indexz paths.npydataimagesdevicecpu) ASSETS_URLz( not found. Downloading images.zip from z /images.zip) safe_downloadT)urlunzipretryz clip:ViT-B/32)rN)r r ultralytics.nn.text_modelrr __import__r faiss_index data_path_npyrgetdata_dirr rexistsultralytics.utilsrrwarningultralytics.utils.downloadsrmodelindex image_pathsload_or_build_index)selfrrrrs l/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/ultralytics/solutions/similarity_search.py__init__zVisualAISearch.__init__1se^^^m^^^^^y>>>>>>;'''(( ((VZZ99:: #FJJx$?$?@@ }##%% + 4 4 4 4 4 4 NdmllU_lll m m m A A A A A A M888A N N N N NNDM%%odkJJJ     """""pathr np.ndarrayc|jtj|S)z7Extract CLIP image embedding from the given image path.)r, encode_imageropendetachrnumpy)r0r4s r1extract_image_featurez$VisualAISearch.extract_image_featureNsFz&&uz$'7'788??AAEEGGMMOOOr3textstrc|j|j|gS)z6Extract CLIP text embedding from the given text query.)r, encode_texttokenizer9rr:)r0r<s r1extract_text_featurez#VisualAISearch.extract_text_featureRsNz%%dj&9&94&&A&ABBIIKKOOQQWWYYYr3cBt|jr~t|jrXt jd|j|j|_tj |j|_ dSt jdg}|j D]}|jdt"vr6 ||||j |j#t*$r)}t jd|jd|Yd}~d}~wwxYw|st/dtj|d}|j||j|jd |_|j||j|j|jtj|jtj |j t jd tC|j d dS) aeLoad existing FAISS index or build a new one from image features. Checks if FAISS index and image paths exist on disk. If found, loads them directly. Otherwise, builds a new index by extracting features from all images in the data directory, normalizes the features, and saves both the index and image paths for future use. zLoading existing FAISS index...Nz#Building FAISS index from images....z Skipping z: z'No image embeddings could be generated.float32zIndexed z images.)"rr$r(r%rinfor read_indexr-nploadr.r'iterdirsuffixlowerlstriprappendr;name Exceptionr* RuntimeErrorvstackastype normalize_L2 IndexFlatIPshapeadd write_indexsavearraylen)r0vectorsfilees r1r/z"VisualAISearch.load_or_build_indexVsd  ! ! ( ( * * tD4F/G/G/N/N/P/P  K9 : : :..t/?@@DJ!wt'9::D  F  9:::M))++ = =D{  ""))#..kAA =t99$??@@@ '' 2222 = = =;49;;;;<<<<<<<< = JHII I)G$$++I66 (((Z++GM!,<==  w tz4+;<<< "BHT-=$>$>??? >s4#344>>>?????s AE FFF皙?querykintsimilarity_threshfloat list[str]c |d}j|j||\ } fdt |dD}|ddtj d|D] \}}tj d|d |d !d |DS) alReturn top-k semantically similar images to the given query. Args: query (str): Natural language text query to search for. k (int, optional): Maximum number of results to return. similarity_thresh (float, optional): Minimum similarity threshold for filtering results. Returns: (list[str]): List of image filenames ranked by similarity score. Examples: Search for images matching a query >>> searcher = VisualAISearch(data="images") >>> results = searcher.search("red car", k=5, similarity_thresh=0.2) rDcg|]?\}}d|kj|td|f@Sr)r.re).0idxiDr0rds r1 z)VisualAISearch.search..sY   8>QYZ[\Y]^aYbfwYwYwT a %!S "2"2 3YwYwYwr3rc|dS)NrE)xs r1z'VisualAISearch.search..s 1Q4r3T)keyreversez Ranked Results:z - z | Similarity: z.4fcg|] }|d Srirp)rjrs r1rnz)VisualAISearch.search..s&&&!&&&r3) rArSrrTr-search enumeratesortrrF) r0rarbrd text_featr-resultsrOscorerms ` ` @r1rwzVisualAISearch.searchs --e44;;IFF   ***:$$Y225      BKERSHBUBU      666 '(((" A AKD% K?t??E??? @ @ @ @&&g&&&&r3c,||S)z.Direct call interface for the search function.)rw)r0ras r1__call__zVisualAISearch.__call__s{{5!!!r3N)rrrr)r4rrr5)r<r=rr5)rr)r_r`)rar=rbrcrdrerrf)rar=rrf) __name__ __module__ __qualname____doc__r2r;rAr/rwr~rpr3r1rrs8####:PPPPZZZZ*@*@*@*@X'''''>""""""r3rc.eZdZdZddd Zdd ZdddZdS) SearchAppaA Flask-based web interface for semantic image search with natural language queries. This class provides a clean, responsive frontend that enables users to input natural language queries and instantly view the most relevant images retrieved from the indexed database. Attributes: render_template: Flask template rendering function. request: Flask request object. searcher (VisualAISearch): Instance of the VisualAISearch class. app (Flask): Flask application instance. Methods: index: Process user queries and display search results. run: Start the Flask web application. Examples: Start a search application >>> app = SearchApp(data="path/to/images", device="cuda") >>> app.run(debug=True) rNrr=r str | Nonerrc8tdddlm}m}m}||_||_t |||_|tdt| d|_ |j d|j d d g d S) zInitialize the SearchApp with VisualAISearch backend. Args: data (str, optional): Path to directory containing images to index and search. device (str, optional): Device to run inference on (e.g. 'cpu', 'cuda'). z flask>=3.0.1r)Flaskrender_templaterequest)rr templatesz/images)template_folder static_folderstatic_url_path/GETPOST) view_funcmethodsN) r flaskrrrrsearcherrrresolveapp add_url_ruler-)r0rrrrrs r1r2zSearchApp.__init__s >***9999999999. &D@@@ 5 't**,,..%     cTZ%QQQQQr3cg}|jjdkrG|jjdd}||}|d|S)zCProcess user query and display search results in the web interface.rrazsimilarity-search.html)r{)rmethodformr&striprr)r0r{ras r1r-zSearchApp.indexsh < & ( (L%))'266<<>>EmmE**G##$rs(#""""" ......33333333777777BBBBBBBB%+ !"O"O"O"O"O"O"O"O"d5"5"5"5"5"5"5"5"5"5"r3