Cutoff length in continual pre-training in Llama-factory?

[NguyenNhoTrung]

When I set cutoff_len = 8192, what will happen to samples that are longer than cutoff_len?
Case 1: They are truncated to a maximum length of 8192.
Case 2: They are split into multiple samples. For example, a sample with a length of 10,000 will be split into one sample with a length of cutoff_len = 8192 and another sample with a length of 10,000 - 8192 = 1908.
Which case will occur with your llama factory library?

[hiyouga]

See https://huggingface.co/learn/nlp-course/chapter7/6?fw=pt#preparing-the-dataset for the method we adopted in preparing pretraining data

[NguyenNhoTrung]

@hiyouga
I have around 21,669 samples, but when running llama-factory with cutoff_length = 8192, it shows the following data:
[INFO|trainer.py:2128] 2024-07-05 03:21:46,989 >> ***** Running training *****
[INFO|trainer.py:2129] 2024-07-05 03:21:46,989 >> Num examples = 2,108
[INFO|trainer.py:2130] 2024-07-05 03:21:46,989 >> Num Epochs = 1
[INFO|trainer.py:2131] 2024-07-05 03:21:46,989 >> Instantaneous batch size per device = 2
[INFO|trainer.py:2134] 2024-07-05 03:21:46,989 >> Total train batch size (w. parallel, distributed & accumulation) = 32
[INFO|trainer.py:2135] 2024-07-05 03:21:46,989 >> Gradient Accumulation steps = 4
[INFO|trainer.py:2136] 2024-07-05 03:21:46,989 >> Total optimization steps = 66
[INFO|trainer.py:2137] 2024-07-05 03:21:46,991 >> Number of trainable parameters = 7,615,616,512
[INFO|integration_utils.py:750] 2024-07-05 03:21:47,714 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"

Question1: Why is there such a large decrease in the number of samples?
Question2: I have some samples with lengths > 132,708 (which is the maximum size of the Qwen model). It raises a warning. Does this affect the training of the model? and do llama-factory shuffle in pre-training data?

Thank you so much!