Why training neural networks comes with a hefty price tag

by tobiniumu
0 comment


In recent years, deep learning has proven to be an effective solution to many of the hard problems of artificial intelligence. But deep learning is also becoming increasingly expensive. Running deep neural networks requires a lot of compute resources, training them even more.

The costs of deep learning are causing several challenges for the artificial intelligence community, including a large carbon footprint and the commercialization of AI research. And with more demand for AI capabilities away from cloud servers and on “edge devices,” there’s a growing need for neural networks that are cost-effective.

While AI researchers have made progress in reducing the costs of running deep learning models, the larger problem of reducing the costs of training deep neural networks remains unsolved.

Recent work by AI researchers at MIT Computer Science and Artificial Intelligence Lab (MIT CSAIL), University of Toronto Vector Institute, and Element AI, explores the progress made in the field. In a paper titled, “Pruning Neural Networks at Initialization: Why are We Missing the Mark,” the researchers discuss why current state-of-the-art methods fail to reduce the costs of neural network training without having a considerable impact on their performance. They also suggest directions for future research.

Pruning deep neural networks after training

The recent decade has shown that in general, large neural networks provide better results. But large deep learning models come at an enormous cost. For instance, to train OpenAI’s GPT-3, which has 175 billion parameters, you’ll need access to huge server clusters with very strong graphics cards, and the costs can soar at several million dollars. Furthermore, you need hundreds of gigabytes worth of VRAM and a strong server to run the model.

There’s a body of work that proves neural networks can be “pruned.” This means that given a very large neural network, there’s a much smaller subset that can provide the same accuracy as the original AI model without significant penalty on its performance. For instance, earlier this year, a pair of AI researchers showed that while a large deep learning model could learn to predict future steps in John Conway’s Game of Life, there almost always exists a much smaller neural network that can be trained to perform the same task with perfect accuracy.

There is already much progress in post-training pruning. After a deep learning model goes through the entire training process, you can throw away many of its parameters, sometimes shrinking it to 10 percent of its original size. You do this by scoring the parameters based on the impact their weights have on the final value of the network.

Many tech companies are already using this method to compress their AI models and fit them on smartphones, laptops, and smart-home devices. Aside from slashing inference costs, this provides many benefits such as obviating the need to send user data to cloud servers and providing real-time inference. In many areas, small neural networks make it possible to employ deep learning on devices that are powered by solar batteries or button cells.

Pruning neural networks early


商品名倉庫クーポンコードクーポン価格(円)限定回数終了日備考
Xiaomi Redmi 9 6.53 inch 4G Smartphone NFC Global Version – Green 4GB+64GB (NFC Version)HK
3HYP128UMA14104円509月30日3%-5%
Xiaomi Redmi Note 9 Pro 4G Smartphone Snapdragon 720G Octa Core 2.3GHz 6.67 inch 64MP + 8MP + 5MP + 2MP Camera 5020mAh Battery Global Version – White 6GB+128GBHK
HA7TH3X6J326147円502月28日3%-5%
Xiaomi Redmi 9C 4G Smartphone 6.53 inch Media Tek Helio G35 2.3GHz Octa-core 13MP AI Triple Camera 5000mAh Battery Global Version – Gray 2GB+32GBHK
フラッシュセール13019円509月14日3%-5%
DOOGEE N20 Mobilephone Fingerprint 6.3inch FHD Display 16MP Triple Back Camera – Purple Hong KongCN
フラッシュセール13019円509月14日3%-5%
DOOGEE N20 Pro Quad Camera Mobile Phones Helio P60 Octa Core 6GB RAM 128GB ROM Global Version – Space GrayCN
フラッシュセール17359円509月14日3%-5%
Ulefone Power 6 4G Smartphone 6350mAh Battery – Black EU VersionHK
GBULEF7E11113019円509月30日3%-5%
DOOGEE S40 Rugged Mobile Phone 5.5inch 4650mAh MT6739 Quad Core 3GB 32GB Android 9.0 8.0MP IP68 – Orange Hong KongHK
フラッシュセール11934円509月14日3%-5%
KUMI Mi1 2G GSM DCS Mobile Phone with Thermometer Function MTK6261D 2.4 inch 32MB RAM 32MB ROM 1700mAh Battery Global Version – BlackCN
GBKUMI1234339円509月16日3%-5%
Cubot KINGKONG MINI 4G Smartphone 4.0 inch Android 9.0 MT6761 Quad Core 3GB RAM 32GB ROM 13.0MP Rear Camera 2000mAh Battery Global Version – OrangeHK
フラッシュセール10849円509月16日3%-5%
Cubot X19 S 4G Smartphone 5.93 inch Android 9.0 MT6763 Octa Core 4GB RAM 32GB ROM 2 Rear Camera 4000mAh Battery Global Version – BlackHK
フラッシュセール10849円509月16日3%-5%
CUBOT KingKong CS 3G Smartphone Android 10 (Go Edition) 2GB RAM 16GB ROM Face ID IP68 Waterproof Global Version – BlackHK
フラッシュセール10849円509月16日3%-5%
Ulefone Armor X3 3G Smartphone – Black EUHK
フラッシュセール10849円509月16日3%-5%
Ulefone Armor X7 4G Smartphone 5 inch Android 10.0 Helio A20 MT6761VWE 2GB RAM 16GB ROM 2 Rear Camera 4000mAh Battery IP68 IP69K Waterproof Global Version – BlackHK
フラッシュセール10849円509月16日3%-5%
Ulefone Note 8P 4G Phablet 5.5 inch Android 10 Go Edition MT6737VW 2GB RAM 16GB ROM 8MP + 2MP Rear Camera 2700mAh Battery Global Version – BlackHK
フラッシュセール7594円509月16日3%-5%
OUKITEL C19 4G Smartphone Android 10 Go 2GB RAM 16GB ROM 6.49 inch HD + 13.0MP + 0.3MP + 0.3MP Rear Camera 4000mAh Global Version – BlackHK
フラッシュセール9764円509月16日3%-5%
World Premise DOOGEE X95 Android 10 4G-LTE Cellphones 6.52 Display MTK6737 16GB ROM Dual SIM – Green Hong Kong 2+16GBCN
フラッシュセール8679円509月16日3%-5%
HOMTOM HT80 IP68 Smartphone 4G LTE Android 10 5.5 Inch MT6737 Quad Core NFC SmarthoneWireless charger – Original green RAM 2GB ROM 16GB Hong Kong CN
フラッシュセール9764円509月16日3%-5%
BMAX X15 Laptop 15.6 inch Intel Gemini Lake N4100 Intel UHD Graphics 600 8GB LPDDR4 RAM 128GB SSD ROM Notebook – Dark GrayCN
GBBMAX12330379円509月5日3%-5%
Xiaomi Mi Laptop 15.6 inch Pro 2020 Upgrade MX350 Notebook Intel i7-10510U 16GB 1TB SSD Computer – I5-10210U 8G 512G ChinaCN
フラッシュセール136819円509月30日3%
CHUWI AeroBook Pro 13.3 inch Notebook Intel Core m3-8100Y 8GB DDR3 256GB SSD Windows 10 OS – PlatinumHK
フラッシュセール48824円509月16日3%-5%
Jumper EZbook X3 Air Notebook 13.3inch IPS Screen Intle Gemini Lake N4100 8GB DDR4 128GB eMMC 1.1cm Ultra-thin design Laptop – Mocha brown US plug ChinaCN
Q53722CCA51EB00136346円509月6日8%
Xiaomi RedmiBook 16 Laptop Ryzen Edition With AMD Ryzen 4700U/4500U 16.1 Inch Display Type C Charge 512G SSD – R5 8G 512G SSD ChinaCN
フラッシュセール81050円509月30日3%
TECLAST P20HD 10.1-inch 4G Tablet Android 10.0 SC9863A Call Eight-core 1.6GHz 4GB RAM 64GB Bluetooth 5.0 EU – BlackCN
フラッシュセール13561円509月14日3%-5%
Teclast F6 Plus 13.3 inch Convertible Laptop 8GB 256GB 360° Flip-and-fold FHD IPS Display Touch Screen Intel N4100 Chip 38Wh(10000mAh / 3.8V)Battery – Cloudy GrayCN
フラッシュセール40144円509月14日3%-5%
Teclast T30 10.1 inch 4G Tablet MTK Helio P70 Octa-core CPU 4GB RAM + 64GB ROM 8.0MP + 5.0MP Camera 8000mAh Battery 5G + 2.4G Dual-band WiFi – BlackCN
フラッシュセール21699円509月14日3%-5%
CHUWI Hi10X 10.1 Inch Tablet PC Intel Gemini Lake N4100 Quad Core CPU Windows 10 OS 6GB LPDDR4 128GB EMMC with Keyboard Stylus Pen – Gray SETHK
フラッシュセール31464円509月14日3%-5%
CHUWI HeroBook Pro 14.1 inch Laptop Intel Gemini Lake N4000 Intel UHD Graphics 600 8GB LPDDR4 RAM 256GB SSD Notebook – Gray EU PlugHK
フラッシュセール29294円509月14日3%-5%
OPPO Realme Band Smart Bracelet Large Color Screen Motion Tracker 16mm Wrist Strap Heart Rate Monitor IP68 USB Charging Wristband International Edition – BlackHK
flash sale2603円509月17日3%-5%
Philips Audio Philips SHP9500 HiFi Precision Stereo Headset Over-Ear Headphones – Black CN
F52F3846A15EB0017259円509月17日5%
Amazfit Pace Smartwatch Amazfit Smart Watch Bluetooth Notification GPS Information Push Heart Rate Monitor for Android – Red Spain (entrepot EU)CN
V52A6B54835EB0003112円509月17日3%
Xiaomi Mijia Redmi Wireless TV Sound Bar Speaker Bluetooth 5.0 Audio Music Playback Redmi Soundbar for Home Theater – add EU adapter CN
M530ED72C4A270015424円509月17日3%
Xiaomi Outdoor Bluetooth Speaker Stereo IP55 Dustproof Waterproof Dual Microphone Noise Reduction Call Bluetooth 5.0 Sound – Black 3%commissionsCN
U5336DF195DEB0004013円509月17日3%-5%
Xiaomi Outdoor Bluetooth Speaker Stereo IP55 Dustproof Waterproof Dual Microphone Noise Reduction Call Bluetooth 5.0 Sound – Mini 3%commissionsCN
M5370247BE5EB0002603円509月17日3%-5%
NEW 2020 Amazfit Neo Smart Watch Bluetooth Smartwatch 5ATM Heart Rate Tracking 28Days Battery Life For Android IOS Phone – Orange 3%commissionsCN
I533102A896270014990円509月17日3%-5%
Xiaomi Airdots Pro 2 Air 2 Bluetooth Headset Wireless Earphone LHDC Tap Control Dual MIC ENC – Airdots Air 2 6%commssionsCN
フラッシュセール4664円509月17日3%-5%
2020 Xiaomi Airdots Pro 2 SE Earphone Mi True Wireless TWS Bluetooth Dual SBC/AAC Tap Stereo Control Dual MIC ENC 5%commissionsCN
フラッシュセール2927円509月17日3%-5%
Original Xiaomi Mijia New 1080P IP Camera 130 Degree FOV Night Vision 2.4Ghz Dual-band WiFi Xiaomi Home Kit Security Monitor – Standrad China EU Plug 5%commissionsCN
M52F7C9E526270012060円509月17日3%-5%
Aqara WSDCGQ11LM Temperature Humidity Sensor ( Xiaomi Ecosysterm Product )CN
フラッシュセール1149円509月17日3%-5%
Aqara Window Door Sensor ( Xiaomi Ecosystem Product )CN
フラッシュセール1041円509月17日3%-5%
Aqara WXKG11LM Smart Wireless Switch Asia Pacific Version ( Xiaomi Ecosystem Product ) – WhiteCN
フラッシュセール997円509月17日3%-5%
CORN WB05 Bluetooth Call Smart Watch 90 Days Standby 1.2 inch 390 X 390 AMOLED Full Touch Screen 8 Sports Modes IP67 Waterproof Smart Watch – BlackCN
フラッシュセール4339円509月17日3%-5%