# Robust LLM Training Infrastructure at ByteDance

## Abstract

Large language model training has scaled to tens of thousands of GPUs, and keeps expanding. As cluster sizes continually scale, training runs can incur failures, interrupting the run and requiring mitigation. Examples may include NaN values, CUDA errors, and job hangs that challenge stability. High-efficiency LLM training infrastructure requires minimal interruption, fast diagnosis, and strong fault tolerance—we present ByteRobust, a GPU infrastructure management system tailored for robust LLM training. Leveraging LLM training’s parallelism and characteristics, it achieves large-capacity fault tolerance, prompt fault demarcation, and localization via a data-driven approach to ensure continuous efficient training. Deployed on a production GPU platform, ByteRobust sets a new level in training robustness, with 97% effective training time ratio over a three-month training job on 9,600 GPUs.

## AI Summary

- Presented a large-scale workload management system called ByteRobust designed for reliable and efficient training of large language models.
- ByteRobust supported flagship models such as C2, CDANS2, CRM5, and C3D, which were used in products like Doubao, CapCut, and JiMeng.
- Highlighted key challenges in large language model training, including frequent interruptions, implicit failures, and silent data corruption.
- Developed a hierarchical stop-time check system and dual-phase replay to isolate and recover from hardware and software faults.
- Achieved a 97% effective training time ratio (ETTR) by minimizing unproductive time through automated incident handling.
- Discussed the importance of deterministic workloads and the need for in-flight checking to detect silent data corruption more efficiently.

## Transcript

**[00:00:09 – 00:00:13]** Hello everyone, my name is Duquan Song, and I'm

**[00:00:13 – 00:00:16]** working for ByteDanceSeed as a software engineer.

**[00:00:16 – 00:00:18]** Today I'm going to talk about ByteRobust.

**[00:00:18 – 00:00:22]** This is a large-scale workload management system designed

**[00:00:22 – 00:00:27]** to enable reliable and efficient training for large language models.

**[00:00:27 – 00:00:31]** This work is a collaboration work between ByteDanceSeed

**[00:00:31 – 00:00:35]** and the University of Hong Kong.

**[00:00:35 – 00:00:40]** So today I'm going to divide my topic into five key parts.

**[00:00:40 – 00:00:45]** To start with, I'll talk about the models that we support.

**[00:00:45 – 00:00:48]** And secondly, we'll look at the observations and challenges

**[00:00:48 – 00:00:51]** in large language model training.

**[00:00:51 – 00:00:55]** Thirdly, we'll move on to the core design and contributions.

**[00:00:55 – 00:00:58]** Then we'll discuss the architecture benefit.

**[00:00:59 – 00:01:02]** Finally, I'll wrap up with a summary of our

**[00:01:02 – 00:01:05]** experience and insight.

**[00:01:06 – 00:01:10]** Okay, first, I'd like to give you a brief view of the models

**[00:01:10 – 00:01:13]** that ByteRobot supports.

**[00:01:14 – 00:01:17]** As a vital link in the training pipeline, ByteRobot makes

**[00:01:17 – 00:01:21]** the large-scale training of our C-series models easier.

**[00:01:21 – 00:01:27]** This includes those flagship models like C2, CDANS2, and CRM5, and C3D.

**[00:01:28 – 00:01:33]** Based on these models, ByteDance developed a lot of great products,

**[00:01:34 – 00:01:40]** which are famous around the world, like Doubao, CapCut, and JiMeng.

**[00:01:40 – 00:01:45]** With ByteRobust, we allow researchers to focus more on

**[00:01:45 – 00:01:50]** the model architecture rather than the infrastructure troubleshooting.

**[00:01:51 – 00:01:55]** Next, I'd like to focus on our observations and

**[00:01:55 – 00:01:58]** key challenges we faced.

**[00:02:00 – 00:02:04]** And today, large-language models are redefining how

**[00:02:04 – 00:02:05]** we work and live.

**[00:02:05 – 00:02:09]** We're seeing technical giants pouring massive resources

**[00:02:09 – 00:02:13]** into building megadata centers, burning through millions of GPU

**[00:02:13 – 00:02:17]** hours to train advanced AI models.

**[00:02:17 – 00:02:21]** But keeping a production-grade training job stable is a

**[00:02:21 – 00:02:23]** massive headache.

**[00:02:23 – 00:02:26]** Here, we are talking about pre-training runs that last

**[00:02:26 – 00:02:28]** weeks or even months.

**[00:02:28 – 00:02:31]** During that time, you cannot just set it and forget it.

**[00:02:31 – 00:02:35]** You are constantly updating the data mixtures, tweaking algorithms,

**[00:02:35 – 00:02:39]** and performance-optimized features.

**[00:02:39 – 00:02:42]** For instance, the parallelism strategies and the kernel

**[00:02:42 – 00:02:46]** performance are continuously tuned when training transforms

**[00:02:46 – 00:02:50]** from the warm-up stage to the full-scale general stage To achieve

**[00:02:50 – 00:02:54]** optimal MFU and memory efficiency,

**[00:02:54 – 00:02:58]** And the data mixtures can be re-weighted during the enhancement

**[00:02:58 – 00:03:04]** stage to boost LLM specific capabilities such as math, coding,

**[00:03:04 – 00:03:08]** and cross-model understanding.

**[00:03:08 – 00:03:11]** These adjustments are just part of the game.

**[00:03:11 – 00:03:15]** In large-language model training, your code isn't a static script.

**[00:03:15 – 00:03:21]** It is a living thing that evolves alongside with the model's need.

**[00:03:21 – 00:03:24]** This code evolution requires pausing the

**[00:03:24 – 00:03:26]** ongoing job for upgrades.

**[00:03:26 – 00:03:30]** Furthermore, inevitable failures also cause interruptions.

**[00:03:30 – 00:03:34]** As shown in the figure on the right, we have observed

**[00:03:34 – 00:03:38]** that there is a 28 restarts for a single large language

**[00:03:38 – 00:03:42]** model job running on the 1,000 GPU.

**[00:03:43 – 00:03:49]** We also observed that the MFU of this job steadily increased during

**[00:03:49 – 00:03:54]** training or into continuously applied engineering optimization.

**[00:03:57 – 00:04:01]** Beyond many stops and restarts for configuration adjustment,

**[00:04:01 – 00:04:06]** understanding the different failure types is also crucial.

**[00:04:06 – 00:04:11]** We collected the incident data from large-language model jobs

**[00:04:11 – 00:04:16]** over three months and classified the failures into two categories.

**[00:04:16 – 00:04:19]** The first one is the explicit failures.

**[00:04:19 – 00:04:24]** These failures have very clear diagnostic indicators,

**[00:04:24 – 00:04:31]** such as the CUDA error, CPU auto-memory, and HDFS error.

**[00:04:31 – 00:04:35]** And the other one is the implicit failure.

**[00:04:35 – 00:04:38]** These kind of failures are very hard to spot because there

**[00:04:38 – 00:04:47]** are no error messages like job hand or MFU decline or NaN value error.

**[00:04:48 – 00:04:51]** Next, let's look at where we are actually losing

**[00:04:51 – 00:04:54]** time during training.

**[00:04:54 – 00:05:00]** We use the ETTR effective training time ratio as a golden

**[00:05:00 – 00:05:04]** metric for us to check

**[00:05:04 – 00:05:09]** How reliable the model training is and the unproductive

**[00:05:09 – 00:05:13]** time is the key factor that determines how good it is.

**[00:05:13 – 00:05:17]** And the unproductive time can be categorized into three parts

**[00:05:17 – 00:05:22]** when there is a failure, detection, localization, and failover.

**[00:05:23 – 00:05:27]** And for detection localization, some failures are very obvious.

**[00:05:27 – 00:05:30]** The system just throws an error, we check the logs,

**[00:05:30 – 00:05:32]** and we fix it very quickly.

**[00:05:32 – 00:05:35]** But the silent failures are the real killer.

**[00:05:35 – 00:05:39]** Sometimes the job just stops running, and we don't realize

**[00:05:39 – 00:05:43]** it until we notice the training stops for a long time or wait

**[00:05:43 – 00:05:45]** until the watchdog timeout.

**[00:05:46 – 00:05:49]** And localization for these implicit failures can also

**[00:05:49 – 00:05:50]** be time consuming.

**[00:05:50 – 00:05:56]** Moreover, there's the cost of failover, which is recovery.

**[00:05:56 – 00:06:01]** The cost depends on the cluster scheduler, training scale,

**[00:06:01 – 00:06:03]** and checkpointing strategy.

**[00:06:03 – 00:06:07]** Specifically, the checkpoint's frequency and adopted

**[00:06:07 – 00:06:10]** storage backend can determine the checkpoint loading

**[00:06:10 – 00:06:13]** and recomputation overheads.

**[00:06:15 – 00:06:19]** So frequent interruptions and failures cause a huge

**[00:06:19 – 00:06:22]** amount of wasted unproductive time.

**[00:06:22 – 00:06:27]** This is because for every failure, the system will go through

**[00:06:27 – 00:06:32]** a lengthy process of detection, localization, and recovery.

**[00:06:32 – 00:06:37]** So our goal is to achieve robust and efficient large-language

**[00:06:37 – 00:06:41]** model training, which means handling failures efficiently and

**[00:06:41 – 00:06:44]** keeping unproductive time minimal.

**[00:06:44 – 00:06:48]** This poses several key challenges.

**[00:06:52 – 00:06:55]** The first challenge is that we have observed that a job

**[00:06:55 – 00:06:59]** with the same symptoms can be caused by different root causes.

**[00:06:59 – 00:07:01]** To give you a concrete example, let's look at

**[00:07:01 – 00:07:04]** the illegal memory access.

**[00:07:04 – 00:07:06]** In our production environment,

**[00:07:06 – 00:07:12]** We observe that illegal memory access could also be caused by

**[00:07:12 – 00:07:19]** the infrastructure fault, like the faulty high bandwidth memory error.

**[00:07:19 – 00:07:24]** If the user who is working, who doesn't know these contacts,

**[00:07:24 – 00:07:29]** he will spend a lot of time troubleshooting their system

**[00:07:29 – 00:07:32]** on their code box.

**[00:07:32 – 00:07:35]** So this requires to consider both the infrastructure

**[00:07:35 – 00:07:39]** faults and bugs introduced by engineers and researchers

**[00:07:39 – 00:07:42]** when troubleshooting a system.

**[00:07:43 – 00:07:49]** So let's move forward to something more frustrating, the job hands.

**[00:07:49 – 00:07:54]** Unlike the memory crashes, these also have no clear chassis.

**[00:07:54 – 00:08:02]** As you can see on the diagram, when the job hands, the RDMA throughput

**[00:08:02 – 00:08:06]** just goes down simultaneously.

**[00:08:06 – 00:08:09]** And all train of process start simultaneously.

**[00:08:09 – 00:08:14]** It is very difficult to isolate the root cause using traditional tools.

**[00:08:14 – 00:08:19]** And the failures could originate from the hardware issues, like GPU,

**[00:08:19 – 00:08:26]** CPU, memory, or NIC, and to users' code of cloud infrastructure.

**[00:08:27 – 00:08:33]** Then we have the silent killer, silent data corruption.

**[00:08:33 – 00:08:37]** So it is a silent data fault that causes non-determinist

**[00:08:37 – 00:08:42]** errors like non-loss without providing explicit diagnostic logs.

**[00:08:42 – 00:08:46]** It is very insidious because collective communication spreads

**[00:08:46 – 00:08:50]** the corruption cluster-wide, making the original faulting nodes

**[00:08:50 – 00:08:54]** nearly impossible to pinpoint.

**[00:08:54 – 00:08:58]** There is no diagnostics, it is very insidious.

**[00:08:59 – 00:09:05]** Finally, failover operations can introduce significant overheads

**[00:09:05 – 00:09:08]** and add additional uncertainty.

**[00:09:08 – 00:09:12]** During job rescheduling, degraded machines may be reassigned

**[00:09:12 – 00:09:15]** to the large-language model job, potentially causing new

**[00:09:15 – 00:09:18]** failures after restarts.

**[00:09:20 – 00:09:24]** To address these challenges, we developed BioRobust.

**[00:09:24 – 00:09:28]** In this section, I'll explore the core design and the key

**[00:09:28 – 00:09:31]** contributions that make BioRobust an efficient solution for

**[00:09:32 – 00:09:36]** large-scale LLM training workloads.

**[00:09:38 – 00:09:43]** So as you can see, this is the high-level workflow

**[00:09:43 – 00:09:48]** of how bi-robots automate the fault tolerance.

**[00:09:50 – 00:09:55]** We want a large language model training infrastructure

**[00:09:55 – 00:10:00]** that enables highly automated incident handling, minimizing

**[00:10:00 – 00:10:04]** human interventions to support scalable training.

**[00:10:04 – 00:10:10]** And we want to prioritize the rapid fault isolation recovery over

**[00:10:10 – 00:10:13]** precise glucose identification.

**[00:10:13 – 00:10:17]** The reason behind it is because we think exhaustive diagnoses

**[00:10:18 – 00:10:22]** increase downtime and leaves a huge amount of healthy GPU sidles.

**[00:10:22 – 00:10:24]** How do we make it?

**[00:10:24 – 00:10:27]** We combine proactive real-time checks with

**[00:10:27 – 00:10:29]** multiple stop-time checks.

**[00:10:29 – 00:10:33]** Based on the check result, bi-robots will execute in-place

**[00:10:33 – 00:10:39]** real-time attempts to recover from faults and code rollbacks.

**[00:10:39 – 00:10:45]** Now let's take a closer look at the real-time checks.

**[00:10:46 – 00:10:51]** Byrobust performs the continuous health checks across GPU,

**[00:10:51 – 00:10:52]** host, and network level.

**[00:10:53 – 00:10:56]** In addition, it collects the metrics and events from

**[00:10:56 – 00:11:00]** our collaborative teams and our monitors.

**[00:11:00 – 00:11:03]** We use the training metrics like...

**[00:11:03 – 00:11:09]** RDMA traffic and tensor utilization as an indicator for us to tell

**[00:11:09 – 00:11:13]** if the training is keep running.

**[00:11:13 – 00:11:18]** And we use the connection-related events like RDMA, TCP,

**[00:11:18 – 00:11:19]** and NIC and switch.

**[00:11:19 – 00:11:24]** It can help us to understand whether there is any

**[00:11:24 – 00:11:26]** connection-related issue.

**[00:11:26 – 00:11:30]** And we also collected GPU events and host events.

**[00:11:30 – 00:11:34]** And for the logs and exit code events, it could help

**[00:11:34 – 00:11:39]** us to understand whether there is any predefined user code error.

**[00:11:40 – 00:11:43]** So let's go to the simplified diagram on the right.

**[00:11:44 – 00:11:47]** When a job crashed at that time, ByteRobot has

**[00:11:47 – 00:11:50]** collected a lot of events.

**[00:11:51 – 00:11:57]** And based on these events, if there is any suspected fault, ByteRobot

**[00:11:57 – 00:12:00]** will try to evict the fault.

**[00:12:01 – 00:12:08]** That is suspected and try to do a reattempt.

**[00:12:08 – 00:12:14]** If we succeed, we will assume that it is an infrastructure fault.

**[00:12:14 – 00:12:17]** So if there is any predefined user errors, we will try to

**[00:12:17 – 00:12:21]** get a rollback and do a reattempt.

**[00:12:21 – 00:12:26]** And if there is not any suspect, we'll try to jump into the

**[00:12:26 – 00:12:29]** stop time checks.

**[00:12:31 – 00:12:36]** Next, let's look at our hierarchical stop-time checks.

**[00:12:36 – 00:12:43]** This is a bit complicated comparing to the proactive

**[00:12:43 – 00:12:47]** real-time checks in terms of an aspect of the workflow.

**[00:12:47 – 00:12:50]** To demonstrate the full capability of the system, let's look

**[00:12:51 – 00:12:56]** at one of the most common yet frustrating issues, the non-laws.

**[00:12:57 – 00:13:02]** The stop time check is triggered if the real-time check yields no clear

**[00:13:02 – 00:13:06]** aspects or if a real attempt fails.

**[00:13:07 – 00:13:11]** Once either of the conditions is met, by robot kicks off

**[00:13:11 – 00:13:13]** the stop tag checks.

**[00:13:13 – 00:13:18]** This brings us the check pipeline itself.

**[00:13:18 – 00:13:20]** First, we will do a GPU test.

**[00:13:21 – 00:13:25]** We'll check the message logs and run NVIDIA UD tests.

**[00:13:25 – 00:13:31]** So this could help us to rule out if there is any GPU-related issue.

**[00:13:31 – 00:13:33]** And then we do a NICL test.

**[00:13:33 – 00:13:37]** We check the connectivity and the bandwidth to see if we

**[00:13:37 – 00:13:41]** miss our performance expectation.

**[00:13:41 – 00:13:46]** And then we have the bitwise alignment test.

**[00:13:46 – 00:13:51]** It is actually a mini GPT verification shield.

**[00:13:51 – 00:13:54]** Yes, so the reason that we use the bitwise alignment test is

**[00:13:54 – 00:14:02]** for is because there are some SDC issues and when there's a failure

**[00:14:02 – 00:14:09]** or a non-loss error we will try to

**[00:14:09 – 00:14:14]** And run the bitwise alignment test on that report because

**[00:14:15 – 00:14:18]** we maintain a set of the losses.

**[00:14:18 – 00:14:23]** And we can compare our expected losses with the workloads.

**[00:14:23 – 00:14:29]** And if there is any mismatch, we'll regard this node as

**[00:14:29 – 00:14:32]** has an SDC issue.

**[00:14:34 – 00:14:40]** Okay, so after the stop time checks, if there's any suspect,

**[00:14:40 – 00:14:46]** ByteRobot will try to evict the machine and do a real attempt.

**[00:14:46 – 00:14:51]** And if there's no suspect, at this time ByteRobot will think

**[00:14:51 – 00:14:54]** there is not a hardware error.

**[00:14:54 – 00:15:00]** Then it will try to do a real attempt if it succeed.

**[00:15:00 – 00:15:02]** There's no conclusion on this issue.

**[00:15:02 – 00:15:10]** However, if neither of these has succeed, we will do a rollback.

**[00:15:10 – 00:15:17]** However, if the rollback cannot solve the problem, then ByteRobots

**[00:15:17 – 00:15:23]** will jump into the last line of our defense, the DOFace replay.

**[00:15:25 – 00:15:30]** Okay, so let me introduce you to our dual-phase replay.

**[00:15:30 – 00:15:33]** The purpose of developing the dual-phase replay is

**[00:15:34 – 00:15:39]** because we want to precisely locate the machines causing

**[00:15:39 – 00:15:42]** hard-to-diagnose implicit failures, like the non-value

**[00:15:42 – 00:15:46]** error, and how we make it.

**[00:15:46 – 00:15:51]** We partition the training machines orthogonally.

**[00:15:51 – 00:15:55]** We play the job in each group with the same model and data

**[00:15:55 – 00:16:00]** and compare the output to locate the abnormal one.

**[00:16:00 – 00:16:04]** As shown in the figure, say we have 24 nodes running for a job.

**[00:16:04 – 00:16:08]** And there is a SDC node, which is the node 13.

**[00:16:16 – 00:16:24]** So we want to find out the 13-node.

**[00:16:24 – 00:16:32]** So in order to find it out, we will first divide the 24

**[00:16:32 – 00:16:35]** nodes into six groups, or four.

**[00:16:35 – 00:16:40]** We first applied the horizontal partitioning, assigning machines

**[00:16:40 – 00:16:42]** with the same partition ID.

**[00:16:43 – 00:16:47]** That is the machine ID, which is the node rank, divided by

**[00:16:47 – 00:16:50]** 4, rounded down to the same group.

**[00:16:50 – 00:16:57]** Then we applied vertical partitioning and do a replay.

**[00:17:00 – 00:17:06]** By intersecting the 40 groups, in this example is horizontal

**[00:17:06 – 00:17:09]** group 3 and vertical group 1.

**[00:17:10 – 00:17:14]** We can find the note 13 as the 40th note, and then we

**[00:17:15 – 00:17:17]** can do an eviction.

**[00:17:20 – 00:17:24]** So then let's move on to the hand-issue diagonals.

**[00:17:24 – 00:17:29]** In order to minimize the unproductive training time,

**[00:17:29 – 00:17:34]** we use over-eviction with multiple sources to make the best confidence

**[00:17:34 – 00:17:38]** we have to solve the issue.

**[00:17:38 – 00:17:46]** And so since we have, besides the system events like xid

**[00:17:46 – 00:17:51]** on network incident, We use this.

**[00:17:51 – 00:17:55]** We use the stack a lot because we think the stack is what makes the

**[00:17:55 – 00:17:59]** troubleshooting issue efficient.

**[00:18:02 – 00:18:04]** So how we make it?

**[00:18:04 – 00:18:09]** First, we pass the process tree of each worker node and the cluster

**[00:18:09 – 00:18:14]** the stack traces for every trainer process to find the majority

**[00:18:14 – 00:18:19]** of the outliers and over-evit the group containing the outliers.

**[00:18:19 – 00:18:26]** In this example, the majority group shows normal reduced-scattered

**[00:18:26 – 00:18:30]** tensor stacks triggered by gradient synchronization in

**[00:18:30 – 00:18:33]** Optimizer step, while the outliers belong to the same

**[00:18:33 – 00:18:40]** pipeline parallel is the minority.

**[00:18:41 – 00:18:47]** So instead of pinpointing the exact 40 machines from the outliers,

**[00:18:47 – 00:18:51]** we evict this whole PP group.

**[00:18:52 – 00:18:58]** So recovery swiftly in a controlled way is also important to us.

**[00:18:59 – 00:19:02]** For bi-robust, we use three mechanisms.

**[00:19:02 – 00:19:06]** In-place hot update, born-standby, and over-eviction of very

**[00:19:06 – 00:19:08]** fast checkpointing.

**[00:19:08 – 00:19:12]** For in-place hot update, we observe that...

**[00:19:12 – 00:19:18]** There is 17% of the interruptions are from manual restarts for

**[00:19:18 – 00:19:20]** code and config adjustment.

**[00:19:20 – 00:19:23]** There is no need to restart the job that has to go through

**[00:19:23 – 00:19:27]** lengthy process of recue, image preparation, and bootstrap

**[00:19:27 – 00:19:29]** script execution.

**[00:19:30 – 00:19:35]** However, in-place hot update cannot cover the case when we evicted

**[00:19:35 – 00:19:40]** the machine because there is new nodes that will be introduced

**[00:19:40 – 00:19:45]** to the training cluster which will ask a lot more overheads on

**[00:19:45 – 00:19:50]** the scheduling, image preparation, bootstrap script execution.

**[00:19:50 – 00:19:54]** So we maintained a set of pre-checked machines and they

**[00:19:54 – 00:19:58]** are ready to take over immediately.

**[00:19:58 – 00:20:01]** So by this way, we can reduce the time cost on

**[00:20:01 – 00:20:04]** new workers' preparation.

**[00:20:12 – 00:20:17]** To accelerate the recovery, we designed the checkpointing

**[00:20:17 – 00:20:22]** modules to restore from the local neighboring backups,

**[00:20:22 – 00:20:26]** reduce the dependency of remote storage access.

**[00:20:26 – 00:20:29]** We further overlap the checkpointing with training

**[00:20:29 – 00:20:34]** and utilize the idle communication cycles to transfer the backups.

**[00:20:34 – 00:20:40]** This fine-grained scheduling eliminate interference and enables

**[00:20:40 – 00:20:44]** High-frequency checkpointing to reduce recomputation

**[00:20:44 – 00:20:47]** costs during failover.

**[00:20:49 – 00:20:54]** The stack aggregation results in potential over-evict.

**[00:20:54 – 00:20:58]** For instance, for 3D parallelism training,

**[00:20:58 – 00:21:01]** DP or PP groups may be evicted.

**[00:21:01 – 00:21:05]** To prevent catastrophic checkpoint loss, our checkpoint

**[00:21:05 – 00:21:09]** backup strategy distributes the checkpoint shots across

**[00:21:09 – 00:21:15]** parallel groups, ensuring recovery even under over-eviction.

**[00:21:17 – 00:21:22]** Okay, so let's talk about the architecture.

**[00:21:22 – 00:21:26]** The architecture for ByteRobots is shown in the figure.

**[00:21:26 – 00:21:32]** The control plan uses the controller to automate the

**[00:21:32 – 00:21:36]** fault tolerance by interacting with Robust Agent and triggers

**[00:21:36 – 00:21:40]** the Runtime Analyzer to perform stack aggregation when job

**[00:21:40 – 00:21:43]** hands or MFU drops are detected.

**[00:21:43 – 00:21:47]** The data plan execute the workloads for monitoring,

**[00:21:47 – 00:21:51]** Diagnosis and Checkpointing on Demand and On-Demand

**[00:21:51 – 00:21:54]** Stack Trace Capture.

**[00:21:55 – 00:21:58]** Thanks to ByteRobot's automatic handling, we've achieved an

**[00:21:58 – 00:22:03]** ETTR up to 97%, meaning our training cluster is productive

**[00:22:03 – 00:22:06]** nearly all the time.

**[00:22:10 – 00:22:17]** So as GPU hardware advanced rapidly, associated monitoring

**[00:22:17 – 00:22:21]** and diagnostic tools often lag behind, making our fault

**[00:22:21 – 00:22:24]** group causes analysis challenging.

**[00:22:24 – 00:22:28]** To bridge this gap, we introduced application-level heuristics

**[00:22:28 – 00:22:33]** such as dual-phase replay and over-eviction.

**[00:22:33 – 00:22:37]** Which have helped us resolve many real-time issues, though their

**[00:22:37 – 00:22:42]** importance gradually diminished as the diagnostic tools mature.

**[00:22:43 – 00:22:47]** Among implicit failures, silent data corruption is critical,

**[00:22:47 – 00:22:52]** yet often overlooked challenge in scaling large language

**[00:22:52 – 00:22:54]** model training to the next level.

**[00:22:54 – 00:23:00]** While dual-phase replay provides effective isolation,

**[00:23:00 – 00:23:04]** It is time-consuming, and we are actively exploring

**[00:23:04 – 00:23:09]** more efficient techniques to address this problem in the future.

**[00:23:11 – 00:23:15]** Okay, I think that's pretty much of it.

**[00:23:23 – 00:23:25]** All right, we have time for questions.

**[00:23:25 – 00:23:27]** Okay.

**[00:23:30 – 00:23:34]** So it sounds like you have a high dependency on the data

**[00:23:34 – 00:23:35]** coming from your components.

**[00:23:35 – 00:23:37]** What kind of telemetry would you expect from your... Sorry,

**[00:23:37 – 00:23:38]** can you speak louder?

**[00:23:38 – 00:23:40]** Oh, sorry.

**[00:23:40 – 00:23:43]** Since you have such a high dependency on the telemetry coming

**[00:23:43 – 00:23:47]** from your components, what kind of data or telemetry would you expect

**[00:23:47 – 00:23:52]** from your storage devices in order to minimize silent data corruption?

**[00:23:53 – 00:23:55]** Okay, so...

**[00:23:57 – 00:23:58]** So that's a good question.

**[00:23:58 – 00:24:05]** So are you asking what kind of the telemetry that we use to

**[00:24:05 – 00:24:08]** detect the silent data corruption?

**[00:24:08 – 00:24:11]** No, I'm asking, so you know what the problem is.

**[00:24:11 – 00:24:14]** What kind of telemetry, like if we develop the next generation

**[00:24:14 – 00:24:18]** of storage that's AI focused, what kind of data would you want

**[00:24:18 – 00:24:22]** it to give you to help you do this?

**[00:24:22 – 00:24:25]** So specifically for storage?

**[00:24:29 – 00:24:34]** Yeah, so we want the data that can tell us that there's

**[00:24:34 – 00:24:40]** silent data corruption in the fly instead of the stop time checks.

**[00:24:40 – 00:24:43]** Yeah, I think that's the most important one, because

**[00:24:43 – 00:24:48]** the stop time checking will be time consuming.

**[00:24:48 – 00:24:53]** Yeah, sometimes there is, the SDC is, cannot give us

**[00:24:53 – 00:24:56]** a non-error issue, non-value issue.

**[00:24:56 – 00:25:03]** It just doing a, we just get a low spike, which will be.

**[00:25:03 – 00:25:07]** Which will be time-consuming to reproduce this kind of issue.

**[00:25:07 – 00:25:13]** So the in-flight checking is, I think, is the most important.

**[00:25:16 – 00:25:18]** Questions?

**[00:25:19 – 00:25:23]** You mentioned SDC is one of the main culprits here.

**[00:25:23 – 00:25:27]** Can you please give more color on like, as errors happen

**[00:25:27 – 00:25:31]** from different components, whether storage, network,

**[00:25:31 – 00:25:38]** or HBM memory, et cetera, how is SDC finally detected?

**[00:25:38 – 00:25:41]** Is it a direct measure or report or telemetry data from

**[00:25:42 – 00:25:46]** the components, or is it indirect measurement from things like

**[00:25:46 – 00:25:49]** loss function, for example?

**[00:25:49 – 00:25:52]** What's the mechanism?

**[00:25:53 – 00:25:56]** So can you repeat that?

**[00:25:56 – 00:25:58]** I'm not sure I understand that.

**[00:25:58 – 00:26:03]** I guess maybe the first question is, are the SDCs

**[00:26:03 – 00:26:07]** on all components or just the GPU?

**[00:26:07 – 00:26:08]** Yes, that's a good question.

**[00:26:08 – 00:26:16]** So for us, we don't have the...

**[00:26:19 – 00:26:26]** We don't have experience the SDC, which is caused by components

**[00:26:26 – 00:26:28]** other than the GPU.

**[00:26:28 – 00:26:30]** And how do we detect it?

**[00:26:30 – 00:26:33]** Like, is it GPU reporting that, hey, I had SDC?

**[00:26:33 – 00:26:37]** Yeah, that's a good question.

**[00:26:38 – 00:26:44]** So, if there is a non-value error in a large language

**[00:26:44 – 00:26:46]** model training, since the

**[00:26:46 – 00:26:54]** We can make the engineering to be deterministic, so if

**[00:26:54 – 00:26:59]** there is a non-loss, we can assume that it is SDC error.

**[00:26:59 – 00:27:08]** However, if there is a loss spike, or if we found the

**[00:27:08 – 00:27:13]** losses cannot be deterministic after we are resuming the job.

**[00:27:13 – 00:27:17]** We will recognize that there must have been an SDC issue

**[00:27:17 – 00:27:23]** because the losses cannot match with the overlap to recompute.

**[00:27:23 – 00:27:28]** But one very important point here I can speak from experience

**[00:27:28 – 00:27:32]** to is that you need a workload that is actually, as you

**[00:27:32 – 00:27:33]** said, deterministic.

**[00:27:33 – 00:27:35]** So if you have a workload that is deterministic, you

**[00:27:35 – 00:27:37]** know what the loss is going to be.

**[00:27:37 – 00:27:42]** So you know, OK, step 0, loss 0.5, step 1, loss 0.4, and

**[00:27:42 – 00:27:43]** then you can verify that.

**[00:27:43 – 00:27:46]** And if you have GPUs that diverge from that, maybe there's

**[00:27:46 – 00:27:48]** something like that.

**[00:27:48 – 00:27:52]** But you have to be careful that, like,

**[00:27:52 – 00:27:55]** Things like Megatron, or maybe PyTorch, or Nickel, they might

**[00:27:55 – 00:27:58]** need some flags to actually be deterministic.

**[00:27:59 – 00:28:02]** You had a follow-up question?

**[00:28:02 – 00:28:04]** So what's the remedy after this?

**[00:28:04 – 00:28:07]** Is it like you repeat the same job and hope SDC doesn't

**[00:28:07 – 00:28:08]** docker next time?

**[00:28:08 – 00:28:11]** Sorry, what's that?

**[00:28:12 – 00:28:15]** You know, once you have SDC and you have detected, what

**[00:28:15 – 00:28:16]** is the recovery out of it?

**[00:28:17 – 00:28:21]** Is it that you say that, okay, this time SDC happened, next time it

**[00:28:21 – 00:28:26]** won't happen and I'll try it again, or you will eliminate the machine?

**[00:28:29 – 00:28:39]** So, do you mean what we do if there is an SDC issue that we notice?

**[00:28:39 – 00:28:42]** Do you mean that?

**[00:28:46 – 00:28:57]** Okay, so we'll try to make sure that if there is no SDC

**[00:28:57 – 00:29:03]** node in the cluster because we want to make sure the training

**[00:29:03 – 00:29:07]** is confident on every step.

**[00:29:07 – 00:29:11]** So talk to your NVIDIA support person, maybe, also.

**[00:29:11 – 00:29:12]** Sorry.

**[00:29:12 – 00:29:14]** Questions? Yes, more questions?

**[00:29:14 – 00:29:18]** Are you satisfied with the answer?

**[00:29:18 – 00:29:21]** She raised her hand.

**[00:29:22 – 00:29:25]** Yeah, so you talked about your ETTR is 97%.

**[00:29:25 – 00:29:29]** What was it before you implement it by robust?

**[00:29:29 – 00:29:32]** Okay, so,

**[00:29:33 – 00:29:38]** That's a good question, so let me think about it.

**[00:29:39 – 00:29:45]** So I think the ETTR is actually developed after we have the

**[00:29:45 – 00:29:47]** robust, actually.

**[00:29:47 – 00:29:53]** So we don't have the exact data for the ETTR, but we

**[00:29:53 – 00:29:56]** have experienced that.

**[00:29:57 – 00:30:01]** There's a case that we're running 3,000 GPU cards, and

**[00:30:01 – 00:30:08]** we spent nearly one week for troubleshooting where the error is.

**[00:30:10 – 00:30:13]** Yeah, so I think it's a huge improvement, but I

**[00:30:13 – 00:30:16]** don't have the exact data.

**[00:30:21 – 00:30:22]** First off, thank you for the talk.

**[00:30:22 – 00:30:25]** I had a question about dual-phase replay.

**[00:30:25 – 00:30:28]** Are you using the real workload when you do the dual-phase replay,

**[00:30:28 – 00:30:31]** or is it a synthetic workload?

**[00:30:31 – 00:30:39]** Yeah, the dual-phase replay, the group is isolated with each other.

**[00:30:39 – 00:30:44]** We have the, in general, we want to make sure that data

**[00:30:44 – 00:30:48]** and the data and the training workloads is identical with

**[00:30:48 – 00:30:51]** each other, but they're not connected with each other.

**[00:30:51 – 00:30:55]** All right, thank you for the clarification.

**[00:30:56 – 00:30:59]** Do we have other questions?

**[00:31:01 – 00:31:03]** Do you guys have a plan to get this open source?

**[00:31:03 – 00:31:07]** Currently, I think no. Sorry about that.

**[00:31:17 – 00:31:22]** Could you talk a little bit more about how you do checks for quality

**[00:31:22 – 00:31:27]** of service, like throughput, latency, network congestion?

**[00:31:27 – 00:31:31]** Could you kind of say where your threshold is there and

**[00:31:31 – 00:31:33]** how you go after that?

**[00:31:33 – 00:31:39]** Yes, so let me go back to the stop time check.

**[00:31:41 – 00:31:49]** Okay, so actually, since our production environment is

**[00:31:49 – 00:31:55]** kind of different because we have multiple data centers with

**[00:31:55 – 00:32:02]** multiple different types of GPUs and NICs, the criteria changes.

**[00:32:02 – 00:32:07]** I think it depends on the workloads.

**[00:32:07 – 00:32:16]** So the basic performance that we think is like, if the, like for the

**[00:32:16 – 00:32:21]** Nikon test, we need to make sure the bandwidth is above like 80% or

**[00:32:21 – 00:32:23]** 70% of the theoretical throughput.

**[00:32:30 – 00:32:36]** For the EUD test, we have a timeout to check if there is

**[00:32:36 – 00:32:39]** any hardware or GPU-related issue.

**[00:32:40 – 00:32:43]** But for the number, I'm not quite sure.

**[00:32:43 – 00:32:48]** I think I can check it with you offline.

**[00:32:50 – 00:32:53]** All right, last chance for questions.

**[00:32:54 – 00:32:55]** Well, I have a question.

**[00:32:55 – 00:32:58]** So what GPU types do you have right now?

**[00:32:59 – 00:33:03]** The primary is the H20, actually, in China.

**[00:33:03 – 00:33:05]** You know that.

**[00:33:05 – 00:33:07]** But you have others, too?

**[00:33:07 – 00:33:08]** H20.

**[00:33:08 – 00:33:11]** But you have other things? Just H20?

**[00:33:11 – 00:33:14]** An H800. Oh, OK.

**[00:33:14 – 00:33:16]** Do you see any difference in the failure modes

**[00:33:16 – 00:33:18]** between the GPU types?

**[00:33:18 – 00:33:23]** Yeah, so that's a good question, let me think.

**[00:33:23 – 00:33:31]** The difference for us is H20 has a higher bandwidth, a

**[00:33:31 – 00:33:33]** higher memory, GPU memory.

**[00:33:33 – 00:33:38]** But in terms of the reliability, I don't see the difference.