[Feature Proposal] Distributed Training for FL #330

Activity

1. slyviacassell

   changed the title [-]Does FedLab support DDP of pytorch?[/-] [+]Does FedLab support multiple GPUs?[/+] on Aug 16, 2023
2. slyviacassell

   changed the title [-]Does FedLab support multiple GPUs?[/-] [+]Does standalone mode support multiple GPUs?[/+] on Aug 16, 2023
3. dunzeng commented on Aug 16, 2023

   

   dunzeng

   on Aug 16, 2023

   Collaborator

   More actions

   We didn't provide multiple GPUs in the standalone module.
   However, you can use the DP module of PyTorch in train function in SGDSerialClientTrainer.
4. slyviacassell

   changed the title [-]Does standalone mode support multiple GPUs?[/-] [+][Feature Proposal] Distributed Training for FL[/+] on Aug 24, 2023
5. slyviacassell commented on Aug 24, 2023

   

   slyviacassell

   on Aug 24, 2023

   Last edited by slyviacassell

   Author

   More actions

   We define the following variables to further illustrate the idea:

   • K: the number of clients who participated in training each round
   • N: the number of available GPUs

   When K == N, each selected client is allocated to a GPU to train.

   When K > N, multiple clients are allocated to a GPU, then they execute training sequentially in the GPU.

   When K < N, you can adjust to use fewer GPUs in training.

   We need to set the number of GPUs in gpu and specific distributed settings in the distributed configs.

   The implementation is under working. Anybody would like to help?

   👍React with 👍1Reacted by Siqi Liang
6. dunzeng

   added

   pending

   on Oct 31, 2023
7. QiTianyu-0403 commented on Sep 28, 2024

   

   QiTianyu-0403

   on Sep 28, 2024

   More actions

   We define the following variables to further illustrate the idea:

   • K: the number of clients who participated in training each round
   • N: the number of available GPUs

   When K == N, each selected client is allocated to a GPU to train.

   When K > N, multiple clients are allocated to a GPU, then they execute training sequentially in the GPU.

   When K < N, you can adjust to use fewer GPUs in training.

   We need to set the number of GPUs in gpu and specific distributed settings in the distributed configs.

   The implementation is under working. Anybody would like to help?

   I'm very interested in the function you mentioned. Is there any code available that can implement this function?