Bart Wronski 🇺🇦🇵🇸Haters gonna hate and fill their hearts with schadenfreude, but AI products ARE delivering huge value (not just for the users, but the companies) and revenue boosts.
Krzysztof Narkowicz@BartWronski Some time ago WSJ reported 20$/user loss: https://archive.ph/3WDSv so it would be interesting to see how Copilot margins look like nowadays.
@k_narkowicz something I heard from a Microsoft director - that they optimized the inference costs by over 400x. Optimizations, better hardware utilization, hardware generations improvements, quantization, mixture of experts models, early outs (have a super cheap model give an answer and its confidence first, if it's low, run the better one).
Which does not surprise me at all, my experience on NTC was also going from multiple hours compression in PyTorch to under a minute in our CUDA version.