Wj; ddlZddlmZddlZddlmZejeZdZdZ Gdde Z dZ d e ejeejd fzd e ejeejd fzfd Z ddedededeeeffdZ ddedededeee effdZeGddZdS)N) dataclass)fxcZgfd}tj||}|fS)z\ Flatten the args into a list form and detach the tensors from computational graph. ct|tjrC||j}||S||SN) isinstancetorchTensordetachrequires_grad_ requires_gradappend)avalflat_detached_argss h/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/torch/distributed/pipelining/_utils.pyextract_tensor_argsz0flatten_args_detach..extract_tensor_argssf a & & ((**++AO<.extract_tensor_args,srr)rrrs @r flatten_argsr&sMI G  rceZdZdZdS)PipeliningShapeErrorz5Shape mismatch between configured and runtime values.N)__name__ __module__ __qualname____doc__rrr r 9s????rr c|j|jks!t|d|jd|j|j|jks!t|d|jd|j||ks;t|d|d|dS)Nz has a shape mismatch: expected z actual z has a dtype mismatch: expected z! has a stride mismatch: expected )shaper dtypestride)descexpectedgivens rvalidate_tensor_metadatar-=s >U[ ( (" Z ZX^ Z ZU[ Z Z    >U[ ( (" Z ZX^ Z ZU[ Z Z    ??    . ." a ahoo6G6G a aQVQ]Q]Q_Q_ a a    / .rexpected_tensors.actual_tensorsc ,t|t|kr2t|dt|dt|dtt|D]$}t|d|||||%dS)Nz: Number of values (z") does not match expected number ()z: value )lenr ranger-)r*r.r/is rvalidate_tensors_metadatar5Ls  N 3 333" x x^)<)< x x`cdt`u`u x x x   3'(( ) )    Q "21"5~a7H      rlooppp_size num_stagesstylereturnc8i}|dkrt|D] }||z||< nv|dkr]||zdkrtd|d|dd}t|D]+}|||<|dz|zdkr||zdzdkr|dz }&|dz},ntd |d |S) z Compute the stage id to rank mapping for either a looped or V-style schedule. Most commonly num_stages == pp_size * 2, but this function can be used to compute the mapping for any number of stages per rank. r6vrz num_stages z% must be evenly divisible by pp_size z for V scheduleszStyle z is not supported.)r3 ValueError)r7r8r9mapping stage_index rank_indexs rgenerate_stage_to_rank_mappingrC[sG  ,, 9 9K#.#8GK  9 #  1 $ $hjhhwhhh   ,,  K#-GK a7*a//w&!+q00a a  ;%;;;<<< Nrct|||}i}|D])\}}||vrg||<|||*|D]}||S)a Compute the rank to stage id mapping for either a looped or V-style schedule. This function inverts the stage_to_rank_mapping to get which stages are assigned to each rank. Returns a dictionary mapping rank -> list of stage indices assigned to that rank. )rCitemsrvaluessort)r7r8r9 stage_to_rankrank_to_stagesstage_idrankstagess rgenerate_rank_to_stage_mappingrM}s37JNNM,.N'--//..$ ~ % %#%N4 t##H----!'')) rc<eZdZUdZejed<eed<eed<dS)PipeInfoz> Captures information for a pipeline (`Pipe` object). graphr8has_loss_and_backwardN) r!r"r#r$rGraph__annotations__intboolr%rrrOrOs> 8OOOOOOrrO)r6)logging dataclassesrr r getLoggerr!loggerrr RuntimeErrorr r-listr tupler5rTstrdictrCrMrOr%rrr_s!!!!!!   8 $ $(((0&@@@@@<@@@      5<(5s1B+CC  &u|S/@)AA     17 !*- #s(^F17 !*- #tCy.4            r