Wj!k UddlZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z m Z ddl Z ddlmZddlmZddlmZmZddlmZdd lmZdd lmZmZmZmZmZmZm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)dd l*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1dd l2m3Z3ddl4m5Z5ddl6m7Z7ej8e9Z:ej;eGdde$Z?Gdde?Z@ d.deAeBe fdedeCde#fdZDdeEe#deEe#fd ZFdeAeBe fd!eCde&fd"ZG d.deEe&d#eCdeHeEe&effd$ZIdedefd%ZJd&ed'edeCfd(ZKd)e jLd*edeCfd+ZMd,eEe&dedeCfd-ZNdS)/N) bisect_rightinsort)ChainMap)Anycast)narrow_tensor_by_index)dedup_save_plans)FLATTEN_MAPPINGflatten_state_dict)_flatten_sharded_tensors) set_element)BytesStorageMetadataChunkStorageMetadataMetadata MetadataIndexSTATE_DICT_TYPE STORAGE_TYPES StorageMetaTensorStorageMetadata)LoadPlan LoadPlannerReadItemSavePlan SavePlanner WriteItem WriteItemType)_compare_save_plans_contains_usable_plan"_create_default_metadata_only_plan_create_read_items_create_write_items_init_state_dict_merge_delta_local_plans)find_state_dict_object)DTensor)_versionlogger)DefaultSavePlannerDefaultLoadPlannercreate_default_local_load_plancreate_default_global_load_plancreate_default_local_save_plancreate_default_global_save_planc eZdZUeed< ddedededzded ed df d Z d d ed edzded dfdZ d e fdZ de e d e e fdZ de e d ee e effdZde e d ee e e e effdZde e d ee e effdZde d e fdZde d e fdZded ejejzfdZded efdZdedefdZdS)!r)mappingsTNFr flatten_sharded_tensorsdedup_replicated_tensorsdedup_save_to_lowest_rankenable_plan_cachingreturnc||_||_i|_||_|td|jj|_||_ dS)NzDefaultSavePlanner's `dedup_replicated_tensors` argument is being deprecated, and no longer has any effect. Please remove this argument from your call.) r r1r0r3r(warning __class____name___cached_plans_key_enable_plan_caching)selfr r1r2r3r4s q/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/torch/distributed/checkpoint/default_planner.py__init__zDefaultSavePlanner.__init__Isa#5'>$ )B& # / NN"    '+n&=$7!!! state_dict storage_metais_coordinatorc|jrt|\}|_|jrt|}||_||_dSN)r r0r1r r@rB)r<r@rArBs r=set_up_plannerz!DefaultSavePlanner.set_up_planner^sQ  " G(::(F(F %J  ' >1*==J$,r?ct|j|j}|jrt j||j}||_|jrw|j tj vrPt|tj |j r+tdtgdS|tj |j <|jS)N planner_datazINo change in the local plan. Skipping sending the plan to the coordinatorFusable)r-r@rBr dataclassesreplacer0planr;r:r_cached_save_planrr(infor)r<rMs r=create_local_planz$DefaultSavePlanner.create_local_planks-dot?RSS  " I&t$-HHHD  $ M&+*GGG'+78NOH  _ 51111HL -d.DEyr? all_plansc,t||jSrD)r r3)r<rQs r=_dedup_save_plansz$DefaultSavePlanner._dedup_save_planss 4+IJJJr?c||}t|\}}|jr8d|D}tt |}t j||}t||std||fS)Ncg|] }|j SrG).0ps r= z:DefaultSavePlanner._create_global_plan..s E E EA E E Er?rGzFailed to validate global plan) rSr.r dictrrKrL_validate_global_plan ValueError)r<rQ deduped_plans global_planmetadataplanner_data_dictmerged_mappingss r=_create_global_planz&DefaultSavePlanner._create_global_plans..y99 ? N N X  " S !F E E E E "8->#?@@O"*8/RRRH$[(;; ?=>> >H$$r?cg}|jtjvrY|tj|j<||\}}|tj|j<|tj|j<|||fSt |sQtgdgt|z}tj|j}tj|j}nttj|j|}|tj|j<||\}}|j|jvruttj|j|D]O\}}t||r%| tgd:| |P|tj|j<|tj|j<|||fS)zw Create global plan with caching. Returns a tuple of global_plan_delta, global_plan, metadata. FrI) r:r_cached_all_plansrb_cached_global_plan_cached_metadatarrlenr#ziprappend)r<rQglobal_plan_deltar^r_ merged_plans cached_plannew_plans r= _create_global_plan_with_cachingz3DefaultSavePlanner._create_global_plan_with_cachings-/  !)F F F ENK )$*@ A$($<$rrrErrPlistrStuplerrbrnrprurwrrTensorrrr~rrrzr|rVr?r=r)r)Fs}$((,04*/$) 88 8"&8#'+ 8 $( 8 " 8 88880,0$ - -# -"D( - -  - - - -8.K4>Kd8nKKKK%h% tH~x' (%%%%*98h98 tH~tH~x7 898989898v0h0 tH~x' (0000,(xH9y9U\BJ5N9999>=>S>>>>9cr?r)c ,eZdZUdZeed<eed< ddededed d fd Z dd ed e d zded d fdZ d e fdZ de e d e e fdZde d e fdZdedejd d fdZdefdZdedejd d fdZded ejfdZdedejfdZd S) r*ak DefaultLoadPlanner that adds multiple features on top of LoadPlanner. In particular it adds the following: flatten_state_dict: Handle state_dict with nested dicts flatten_sharded_tensors: For FSDP in 2D parallel mode allow_partial_load: If False, will raise a runtime error if a key is present in state_dict, but not in the checkpoint. original_state_dictr0TFr r1allow_partial_loadr5NcL||_||_i|_i|_||_dSrD)r r1rr0r)r<r r1rs r=r>zDefaultLoadPlanner.__init__s1 #5'>$#%  "4r?r@r_rBct|||_|jrt|}|jrt |\}|_||_||_||_dSrD) r"rr1r r r0r@r_rB)r<r@r_rBs r=rEz!DefaultLoadPlanner.set_up_planner)sn $$$#-  ' >1*==J  " G(::(F(F %J $  ,r?c|jtd|jrt|j}t|jj}||z }|rddt_t|j \}}t|}||zr||c|_|_ dt_t|j|j|j S)Nzself.metadata is not None2_3) r_AssertionErrorr setr@keysstate_dict_metadatar'_derived_versionrr0r+r)r< current_keys load_keys missing_keysold_state_dict old_mappingsold_keyss r=rPz$DefaultLoadPlanner.create_local_plan<s =  !<== =  " 1$t335566LDM=BBDDEEI$|3L 1,1)/A,00, ~224455l*R5C\2DOT]-1)- OT]0G,G   r?r^c t|SrD)r,)r<r^s r=rpz%DefaultLoadPlanner.create_global_plands.{;;;r?rmc|SrDrV)r<rms r=rwzDefaultLoadPlanner.finish_plangsr? read_itemvaluec|jrAt|j|j|jjt j|ddSt j|d|j|jj<dS)NF) weights_only) r r rr0 dest_indexfqnrloadr@)r<rrs r= load_byteszDefaultLoadPlanner.load_bytesjs  "  ( i267 5u555      9> E999DOI04 5 5 5r?cb||j}|||SrD) lookup_tensorrtransform_tensorr<rtensors r=resolve_tensorz!DefaultLoadPlanner.resolve_tensorvs.##I$899$$Y777r?rcdSrDrVrs r= commit_tensorz DefaultLoadPlanner.commit_tensorzs r?r{c,t|j|Srrrs r=rz DefaultLoadPlanner.lookup_tensor}rr?c8t||j|jSr)r dest_offsetslengthsrs r=rz#DefaultLoadPlanner.transform_tensors%fi.DiFWXXXr?)TTFr)r9rr__doc__rrr rr>rrErrPrrprwrrrrrrrrrrrrVr?r=r*r*s)((($((,#( 5 5  5"& 5! 5  5 5 5 5%)$ --#-T/- -  ----&& 8& & & & P=>U\>>>>Y(YELYYYYYYr?r*c `eZdZdZd fd ZdededefdZ d d e dedzd eddffd Z xZ S)_EmptyStateDictLoadPlannera Extension of DefaultLoadPlanner, which rebuilds state_dict from the saved metadata. Useful for loading in state_dict without first initializing a model, such as when converting a DCP checkpoint into a Torch save file. . N.B. `state_dict` must be an empty dictionary when used with this LoadPlanner .. warning:: Because the entire state dict is initialized, It's recommended to only utilize this LoadPlanner on a single rank or process to avoid OOM. NcH||_tj|i|dSrD)rsuperr>)r<rargskwargsr8s r=r>z#_EmptyStateDictLoadPlanner.__init__s, $)&)))))r?keyr_r5c VjdS|jvrdSg}|j|}|D]W}|r>|d|dt |gB||Xt fd|DrdSdS)NT.c3*K|] }|jvVdSrD)r)rWunflattened_keyr<s r= zA_EmptyStateDictLoadPlanner._should_include_key..s*TT$)+TTTTTTr?F)rrHgetrijoinstrany)r<rr_unflattened_keysrHrs` r=_should_include_keyz._EmptyStateDictLoadPlanner._should_include_keys 9 4 $)  4&(,0055 + 9 9O 9 ''HH.r2C4H4HIJJ !''8888 TTTTCSTTT T T 4ur?Fr@rBc|rtd|td|jD]\}}|||st |t r%t j|j|j j }|j &||j vrt||j |||||<t|||dS)Nznot state_dictzmetadata is not None)dtype)rritemsr isinstancerremptysize propertiesrrHr rrE)r<r@r_rBkvr8s r=rEz)_EmptyStateDictLoadPlanner.set_up_planners  3 !122 2   !788 806688 " "DAq++Ax88 !233 BKal.@AAA$0Q(:O5O5OJ(=a(@!DDDD ! 1  z8^DDDDDr?rDr) r9rrrr>rrrrrrE __classcell__)r8s@r=rrs  ******sh44%)$ EE#ET/E E  EEEEEEEEEEr?rTr@r_strictr5c Dg} |D]\}}||jvr|rtd|d$|j|}t|tr]t |ddL|j|kr/td|jd|d|t|tr.|j |t|||z }|t|||z }t|S)Nz&Missing key in checkpoint state_dict: rrzSize mismatch between saved z and current: z for ) rr RuntimeErrorrrgetattrrr\r% device_meshget_coordinater r)r@r_rrequestsrobjmds r=r+r+sHH$$&&99S h2 2 2 "#RC#R#R#RSSS  )# . r0 1 1 VT**6388::%%\rw\\chhjj\\WZ\\  c7 # # 9--//;.sB<<< *3C88 8HH H  r?rQc|S)z Create global load plan used by DefaultLoadPlanner. The default load behavior involved no global coordination and this function currently doesn't change the local plans. rV)rQs r=r,r,s  r?rBcg}|D]Z\}}t|tr-|j|t ||z }G|t ||z }[t |S)a Create the ``SavePlan`` used by DefaultSavePlanner. On non-coordinator ranks, this function ignores tensors and non-tensor objects, only producing writes for ShardedTensor objects. On the coordinator rank, produce writes for all values. )rrr%rrr!r)r@rBrrrs r=r-r-sH$$&& 6 6S c7 # # 6--//;/S999 +C55 5HH H  r?rewrite_index_hintsc i}g}|D]}g}|jD]}|jtjkr|jj|vrt d|jtjkr1t||jj<| |{|j t dtt| |jjt|j j|j jg}|}|rCt!j|jt%|j} t!j|| }| ||j jt d|jjd|j |j j| t!j|||t+|fS) a6 Create the global plan and metadata used by DefaultSavePlanner. Metadata is produced by concatenating the metadata of all ``WriteItem`` from the supplied plans. The only global planning change is to update index hints in all ``MetadataIndex`` objects if ``rewrite_index_hints`` is True. zitem.index.fqn not in mdNzitem.tensor_data is not None)rrchunks)r{zZ Cannot create MD for tensor without bounds. FQN: z )r)rrrSHARDr{rrrrri tensor_datarr setdefaultrrrKrLrgrchunkr) rQrr new_plansrM new_itemsitem tensor_mdnew_item new_indexs r=r.r.s$&BI&E&E J# @# @DyM///:>R''()CDDDyM111%9%;%;4:>"  &&&&#+()GHHH )MM -'+'7'B!%!1!6#%    &J + 3 #i.>*?*?!!!I +24yIIIH  ***#)1(**.*** ''(8(>????,TCCCDDDD x|| $$r?cJt|}t|g\}}|S)zTReturn the ``Metadata`` if DefaultSavePlanner was used to checkpoint ``state_dict``.)rr.)r@rM_rs r=_create_default_local_metadatarRs' -j 9 9D +TF 3 3EAr Ir?box0box1ct|j}t|D]\}|j||j||j|zkrdS|j||j||j|zkrdS]dS)z9Check if two boxes overlap. Tuples are (offset, lengths).FT)rgoffsetsrangesizes)rrndimsis r=_check_box_overlaprYs   E 5\\ <?dl1o 1 = = =55 <?dl1o 1 = = =55 > 4r?outer_box_size inner_boxctt|D]R}|j|dkrdS|j|dkrdS|j||j|z||krdSSdS)NrFT)rrgrr)rrrs r=_check_box_boundsris3~&& ' '  Q ! # #55 ?1  ! !55  Q )/!"4 4~a7H H H55 I 4r?r^c Nd}|jD]\}ttrt jdkr5jd}D]W}tj|s$t d|j|d}|tj |j z }Xt dkrt jttdfdttt fd }g}|D]}|} | j} | | j z} t#|| t$jf} | r|d| =|D]<\} }|}t)| |rt d || |d}=t+|| |ftj j}t |dkr%||krt d |||d}|S) NTrz~ key:%s has out of bounds chunk: tensor-size:%s chunk: %s Fr&cj|SrD)r)drs r=z'_validate_global_plan..s %*Q-r?)defaultrchjgfdtDRS)Nc3>K|]}j|VdSrD)r)rWrridxs r=rz:_validate_global_plan....s/BBfSk)!,BBBBBBr?)rr)r rdims sweep_dims`r=rz'_validate_global_plan..sD3K' 2!BBBBBeDkkBBB!!r?)rz$key:%s has overlapping chunks: %s %szq key:%s invalid fill tensor-volume: %s chunks-volume: %s )rrrrrgrrrr(r7mathprodrmaxrsortedrrsysmaxsizerr)r^r_all_goodr chunks_volumersorted_indicesactiver currentstartendcutoffr other_idxother tensor_volumerr r rs @@@@r=r[r[wsH288::CC U e1 2 2   uz??a     4 4E$UZ77 !J! TYu{33 3MM v;;??uz??DE$KK8O8O8O8OPPPI#c&kk""N-/F% + + + 2gmI66%fuck.BCC(ww$* ) )LAy"9-E)'599)B#!  $)vSz**** %*-- {  a  M]$B$B NN   H Or?)T)OrKrloggingr rbisectrr collectionsrtypingrrrtorch.distributed._shard._utilsr.torch.distributed.checkpoint._dedup_save_plansr )torch.distributed.checkpoint._nested_dictr r 2torch.distributed.checkpoint._sharded_tensor_utilsr &torch.distributed.checkpoint._traverser %torch.distributed.checkpoint.metadatarrrrrrrr$torch.distributed.checkpoint.plannerrrrrrrr,torch.distributed.checkpoint.planner_helpersrrrr r!r"r#"torch.distributed.checkpoint.utilsr$torch.distributed.tensorr%r' getLoggerr9r(Loggerr__all__r)r*rrZrrr+rr,r-rr.rrSizerr[rVr?r=r0s  ''''''''  BBBBBBKKKKKKXWWWWW>>>>>>                    FEEEEE,,,,,,+*844444   FFFFFFFFRtYtYtYtYtYtYtYtYn@E@E@E@E@E!3@E@E@EHDH''S#X'*2'<@' ''''T H~  (^    S#X04 :!%5%5%H~5%5% 4>8 #$5%5%5%5%p8 1 9M RV     J +?      GtH~GGdGGGGGGr?