Open Daily: 10am - 10pm | Alley door pickup 10am - 7pm 3038 Hennepin Ave Minneapolis, MN
612-822-4611

  • Open Daily: 10am - 10pm
    Alley-side Pickup: 10am - 7pm

    3038 Hennepin Ave Minneapolis, MN
    612-822-4611

Open Daily: 10am - 10pm | Alley-side Pickup: 10am - 7pm
3038 Hennepin Ave Minneapolis, MN
612-822-4611
Parallel Computing Cluster for Solving Computational Problems in Data

Parallel Computing Cluster for Solving Computational Problems in Data

Mutyala Rao, Chavala

Paperback

General Computers

ISBN10: 6209340733
ISBN13: 9786209340734
Publisher: LAP Lambert Academic Publishing
Published: Dec 12 2025
Pages: 56
Weight: 0.19
Height: 0.13 Width: 6.00 Depth: 9.00
Language: English
Thisbook tackles a critical bottleneck in large-scale AI: the slow and communication-heavy training of massive Deep Neural Networks (DNNs) on multi-GPU systems. It addresses the trade-off between two main parallelization methods. Data parallelism suffers from severe communication overhead for large models, while pipelined model parallelism (like PipeDream) offers up to 8.91x speedup for large Fully Connected/Recurrent Neural Networks but causes weight staleness, degrading model accuracy. To resolve this, the paper introduces SpecTrain, a novel technique. SpecTrain uses the momentum from optimizers to predict future weight updates, allowing pipelined computation with accurate, non-stale weights. This enables the high GPU utilization and speed of pipelining while maintaining the training robustness and final accuracy of synchronous methods.

Also from

Mutyala Rao, Chavala

View All

Also in

General Computers

View All