how to use 200k model to do SFT about yi HOT 8 CLOSED

A singe node with 8 x A100 80G cards could sft 4k context ?

By our experiments, it is okay to sft 4k with zero2+offload on a single node, and sft 16k with zero3+offload by properly setting deepspeed configs in utils/ds_utils.py

from yi.

A singe node with 8 x A100 80G cards could sft 4k context ?

By our experiments, it is okay to sft 4k with zero2+offload on a single node, and sft 16k with zero3+offload by properly setting deepspeed configs in utils/ds_utils.py

6B or 34B ?

34b

from yi.

200k=200000 in Yi model

from yi.

so just modify this position？

from yi.

change max_seq_len in script args, as well as max_position_embeddings in config.json in model file to 200k. But this will obviously need more GPU memory. A singe node with 8 cards may not be enough.
Consider using multi-node sft training by setting deepspeed -H /path-to-hostfile/hostfile main.py ........

from yi.

A singe node with 8 x A100 80G cards could sft 4k context ?

from yi.

A singe node with 8 x A100 80G cards could sft 4k context ?

By our experiments, it is okay to sft 4k with zero2+offload on a single node, and sft 16k with zero3+offload by properly setting deepspeed configs in utils/ds_utils.py

thanks

from yi.

A singe node with 8 x A100 80G cards could sft 4k context ?

By our experiments, it is okay to sft 4k with zero2+offload on a single node, and sft 16k with zero3+offload by properly setting deepspeed configs in utils/ds_utils.py

6B or 34B ?

from yi.