/jKddlmZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z ddl mZddlmcmZddlmZmZmZmZmZe rddlmZmZehdZeeZdZ d Z!Gd d Z"dKdZ#dddLdZ$dMdZ%dNd!Z& dOdPd)Z' dQdRd0Z(dSd1Z)dTd3Z*dUd6Z+dVd;Z,dWd>Z-dUd?Z.dXdBZ/dUdCZ0dYdEZ1dZdFZ2d[dHZ3d\dJZ4dS)]) annotationsN)chain)IO TYPE_CHECKING)AttributeProto FunctionProto GraphProto ModelProto TensorProto)CallableIterable>lengthoffsetbasepathchecksumlocation dceZdZddZdS)ExternalDataInfotensorr returnNonecd|_d|_d|_d|_d|_t }d}|jD]}|jtvrt||j|j ,|dz }t|tkrK|jdt}t|jtkr|dz }|||rht|}|t|z }t!|}|dkr |d|dz }t#jd|d|jd t(d |jCt+|j|_|jdkrt-d |jd|j|jCt+|j|_|jdkr!t-d |jd|jdSdS)Nrz...z and z morez&Ignoring unknown external data key(s) z for tensor z. Allowed keys: ) stacklevelz/External data offset must be non-negative, got z/External data length must be non-negative, got )rrrrrset external_datakey_ALLOWED_EXTERNAL_DATA_KEYSsetattrvaluelen_MAX_UNKNOWN_KEYS_IN_WARNING_MAX_KEY_DISPLAY_LENGTHaddsortedreprwarningswarnname_SORTED_ALLOWED_KEYSint ValueError) selfr unknown_keysunknown_key_countentry truncatedshownextrakey_lists ^/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/onnx/external_data_helper.py__init__zExternalDataInfo.__init__3s-     !$ ) 0 0Ey777ei5555!Q&!|$$'CCC % *B+B*B CI59~~(???!U*  $$Y///  <((E%E 2EE{{Hqyy0E0000 M888$k88!588      ; "dk**DK{Q 2dk22"(+22 ; "dk**DK{Q 2dk22"(+22 # "N)rr rr)__name__ __module__ __qualname__r:r;r9rr2s(000000r;r data_file IO[bytes]info tensor_namestrrbytesc tj|j}|jB|j|krt d|jd|d|||j|jZ|j|jnd}||z }|j|kr t d|jd|d|d|||jS|S) a Validate offset/length against actual file size and read data. Layer 3 defense-in-depth (CWE-400): prevents memory exhaustion even if the model was crafted via direct protobuf APIs that bypass Python parsing. Returns the raw bytes read from the file. NzExternal data offset (z) exceeds file size (z ) for tensor rzExternal data length (z) exceeds available data (z bytes from offset ) osfstatfilenost_sizerr0seekrread)r@rBrC file_size read_start availables r9#_validate_external_data_file_boundsrPfs/))++,,4I { ; " "<<<<<,7<<  t{### {$(K$;T[[  * ; " ".....2<..)..  ~~dk*** >>  r;T check_existsbase_dir data_pathrRboolc 4tj|}tj|}||tjzs$||krt jd|d|d|dtj|rt jd|d|d|rWtj|r8tj |j dkrt jd|d|d|S) u Validate that an external data path is safe to open. Performs three security checks: 1. Canonical path containment — resolved path must stay within base_dir. 2. Symlink rejection — final-component symlinks are not allowed. 3. Hardlink count — files with multiple hard links are rejected. Args: base_dir: The model base directory that data_path must be contained in. data_path: The external data file path to validate. tensor_name: Tensor name for error messages. check_exists: If True (default), check hardlink count. Set to False for save-side paths where the file may not exist yet. Returns: The validated data_path (unchanged). Raises: onnx.checker.ValidationError: If any security check fails. Tensor z external data path resolves to z& which is outside the model directory .z external data path z? is a symbolic link, which is not allowed for security reasons.rzD has multiple hard links, which is not allowed for security reasons.) rGpathrealpath startswithsep onnx_checkerValidationErrorislinkexistsstatst_nlink)rSrTrCrR real_base real_paths r9_validate_external_data_pathress6  **I  ++I    BF 2 3 3 Y8N8N* Qk Q Q Q QBK Q Q Q    w~~i   * Nk N N N N N    y11 bgi6H6H6QTU6U6U* Sk S S S S S    r;rr rct|}tj||j|j}t j}tt dr|t jz}t j ||}t j |d5}t|||j|_ ddddS#1swxYwYdS)zLoads data from an external file for tensor. Ideally TensorProto should not hold any raw data but if it does it will be ignored. Arguments: tensor: a TensorProto object. base_dir: directory that contains the external data. O_NOFOLLOWrbN) r c_checker_resolve_external_data_locationrr-rGO_RDONLYhasattrrgopenfdopenrPraw_data)rrSrBexternal_data_file_path open_flagsfdr@s r9load_external_data_for_tensorrss F # #D'G$- Jr<  $bm# (* 5 5B 2t   = tV[                    s B33B7:B7modelr ct|D]<}t|r+t||tj|_|jdd==dS)zLoads external tensors into model Arguments: model: ModelProto to load external data to base_dir: directory that contains external data N)_get_all_tensorsuses_external_datarsr DEFAULT data_locationr )rtrSrs r9load_external_data_for_modelrzs`#5))(( f % % ( )&( ; ; ;#.#6F $QQQ' ((r;rr int | Nonerr str | Nonerc|dstd|jzdz|jdd=tj|_||t|nd|t|nd||dD];\}}|4|j }||_ t||_ = the size_threshold it will be converted to external data. To convert every tensor with raw data to external data set size_threshold=0. convert_attribute (bool): If true, convert all tensors to external data If false, convert only non-attribute tensors to external data Raise: ValueError: If location is not a relative path. FileExistsError: If a file already exists in location. z.datazDlocation must be a relative path that is relative to the model path.zExternal data file exists in rXroN)_get_initializer_tensorsrvrDuuiduuid1rGrYisabsr0r`FileExistsErrorr~sys getsizeofrorr-_is_valid_filename) rtrrrrtensors file_namertensor_locations r9convert_model_to_external_datars4'u--G*"5)); %%/  !w}}X&&  Zw~~h'' S%&Qh&Q&Q&QRRR I 5 5F ++ 5M&/22nDD!&)444  5 5 ; ;F ++ ;M&/22nDD"(+)/::8&)$*,,&7&7O!&/::: ; ;r;ct|D]P}t|r?|dstd|jdd=t j|_QdS)zCall to set all tensors which use external data as embedded data. save_model saves all the tensors data as embedded data after calling this function. Arguments: model (ModelProto): Model to be converted. roraw_data field doesn't exist.N)rvrwr~r0r r rxry)rtrs r9 convert_model_from_external_datar9st#5))77 f % % 7??:.. B !@AAA$QQQ'#.#6F 77r; base_pathc t|}tj|j}|r't jd|jd|jd|d|jvr(t jd|jd|d|jd|jdvr%t jd|jd |jd tj ||j}t|||jd | d st jdtjtjz}t!tdr|tjz}tj||d}tj|d5}|dd|jY|}|j|kr |d|j|z z||j|} ||jt3||j| || z ddddS#1swxYwYdS)aWrites tensor data to an external file according to information in the `external_data` field. The function checks the external is a valid name and located in folder `base_path`. Arguments: tensor (TensorProto): Tensor object to be serialized base_path: System path of a folder where tensor data is to be stored Raises: ValueError: If the external file is invalid. rWzC is external and must not be defined with an absolute path such as z , base_path=..z* is external and must be placed in folder z, '..' is not needed in rX)rXrz is external and its name z is invalid.TrQrorrgizr+brrN)rpathlibPathr is_absoluter]r^r-partsrGrYjoinrer~O_CREATO_RDWRrlrgrmrnrKrtellwriteror) rrrB location_pathrprqrrr@rMrs r9save_external_datarIs  F # #DL//M  "" * 'fk ' '-1] ' '" ' '    }"""* Gfk G G G G48M G G G   [((* -fk - -} - - -   !gll9dmDD!*FKd ??: & &L*+JKKKbi'Jr<  $bm# (*e <          r;c#>Kt|jEd{VdS)z:Create an iterator of initializer tensors from ONNX model.N)rrrs r9rrs023C3IJJJJJJJJJJJr;graph_or_functionGraphProto | FunctionProtoc#K|jD]R}|jD]H}|dr |jV|jEd{Vt |t Ed{VISdS)zSCreate an iterator of tensors from node attributes of an ONNX model graph/function.tN)rrr~rrr!_get_attribute_tensors_from_graph)rrrs r9rrs"&  I!!#&& "k!!! ( ( ( ( ( ( ( (5<          r;c#|Kt|jEd{V|jD]}t|Ed{VdS)zDCreate an iterator of tensors from node attributes of an ONNX model.N)rr functions)rfunctions r9rrsi01A1GHHHHHHHHH$.??4X>>>>>>>>>>??r;filenamecrtjd}||}t|S)z8Utility to check whether the provided filename is valid.z^[^<>:;,?"*|/]+$)recompilematchrU)rexprs r9rrs/ *' ( (C IIh  E ;;r;cV|do|jtjkS)z?Returns true if the tensor stores data in an external location.ry)r~ryr r)rs r9rwrws+ (( 9  K$8 8r; field_keyc`t|jD]\}}|j|kr|j|=dS)aRemoves a field from a Tensor's external_data key-value store. Modifies tensor object in place. Arguments: tensor (TensorProto): Tensor object from which value will be removed field_key (string): The key of the field to be removed N) enumerater r!)rrifields r9remove_external_data_fieldrsEf233((5 9 ! !$Q'((r;filepathct|D]K}t|r:|dr%t|||dL|S)aSerializes data for all the tensors which have data location set to TensorProto.External. Note: This function also strips basepath information from all tensors' external_data fields. Arguments: model (ModelProto): Model object which is the source of tensors to serialize. filepath: System path to the directory which should be treated as base path for external data. Returns: ModelProto: The modified model object. ro)rvrwr~r ClearField)rtrrs r9write_external_data_tensorsrsj#5))** f % % *&//**E*E * vx 0 0 0   j ) ) ) Lr;)r@rArBrrCrDrrE) rSrDrTrDrCrDrRrUrrD)rr rSrDrr)rtr rSrDrr)NNNN)rr rrDrr{rr{rr|rr|rr)TNrF) rtr rrUrr|rr/rrUrr)rtr rr)rr rrDrr)rr rr)rrrrrr)rr rr)rrrr)rrDrrU)rr rrU)rr rrDrr)rtr rrDrr )5 __future__rrGrrrrr+ itertoolsrtypingrr onnx.checkercheckerr]onnx.onnx_cpp2py_export.checkeronnx_cpp2py_exportri onnx.onnx_pbrrr r r collections.abcr r frozensetr"r)r.r&r'rrPrersrzrrrrrvrrrrrrrwrrr?r;r9rs'#"""""  $$$$$$$$######333333333322222222 (i<<<v9::!11111111h    P ,,,,,,^    2 ( ( ( (& !!!!!@%)# 7;7;7;7;7;t 7 7 7 7 ?T?T?T?TD####KKKK    ???? ( ( ( (r;