There are very promising new technologies like Thermodynamic Computing (https://medium.com/@thebojda/what-is-thermodynamic-computing-and-how-does-it-help-ai-development-1dd3b75a9ee6), or 1,5 bit quantized models with ASIC (https://medium.com/datadriveninvestor/the-enormous-potential-of-binarized-and-1-58-bit-neural-networks-e4209766f6f3). So, I hope, in the near future training huge networks won't be a privilage of big companies.