Độ trễ LLM không chỉ do kích thước mô hình! Nguyên nhân chính thường nằm ở hạ tầng: hàng đợi yêu cầu, chiến lược batching (nên dùng continuous), bộ lập lịch token và áp lực bộ nhớ (KV cache). Để tối ưu, hãy dùng system prompts, quản lý tốt client-side (giới hạn đồng thời, exponential backoff). Các hệ thống như vLLM, TGI hiệu quả hơn.
#LLM #AI #Latency #Optimization #TốiƯu #ĐộTrễ

https://www.reddit.com/r/LocalLLaMA/comments/1p71cas/hidden_causes_of_llm_latency_its_not_just_the/

Measuring Latency (2015)

https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/

#HackerNews #Measuring #Latency #Latency2015 #TechInsights #NetworkPerformance #DataAnalysis

Our colleague Frank (Frantisek Borsik) will be continuing @mtaht "State of the Bloat" presentation series at #UnderstandingLatency 4.0 by CUJO AI® - join us ONLINE and for FREE! Register at https://understandinglatency.com/

December 15-17, 3:00 - 5:30 PM UTC

#bufferbloat #latency #jitter #DaveTaht #OpenSource #RFC8290 #FQ_CoDel #QoE #sch_CAKE #QualityOfExperience #BandwidthIsALIE #QoS #broadband #WiFi #OpenWrt #Linux #CUJOAI #LowLatency #ISP #WISP #FISP #FWA #FLOSS #FQCoDel #schCAKE #InternetService

Bài trình bày tại #JVMLS khám phá kỹ thuật Off-CPU Profiling & Chẩn đoán độ trễ trong Java, giúp tối ưu hiệu suất ứng dụng.
#Java #Performance #Profiling #Latency #Diagnostics #PhanTichHieuSuat #DoTre #JVMLS

https://www.reddit.com/r/programming/comments/1p0dofd/offcpu_profiling_latency_diagnostics_in_java_jvmls/

#LibreQoS is an example that you can develop a world changing startup from the borderland. We are thankful to KTSM 9 NEWS for highlighting our work on fixing #latency, #bufferbloat & #jitter for Internet Service Providers and their customers around the world:

https://www.ktsm.com/news/el-pasoans-operating-software-startup-designed-to-improve-internet-connectivity/

We would like to dedicate this to our beloved colleague #DaveTaht (1965 - 2025) that was instrumental in the global effort on fixing these issues for the #ISPs & their customer everywhere!

#OpenSource #FLOSS

We just launched version 0 (Alpha) of the #LibreQoSInsight API. #LibreQoS shapers have an API, too — and that’ll be the topic of future posts.

This one is all about the Insight #API:
https://devblog.libreqos.com/posts/0008-insight-api1/

#QoE #latency #bufferbloat #QualityOfExperience #OpenSource #QoS #RFC8290 #FQ_CoDel #sch_CAKE #broadband #ISP #FISP #WISP #FLOSS #QualityOfService #bandwidht #BandwidthIsALIE #FQCoDel #schCAKE #jitter

#LibreQoS interview with Thomas of #TheBrothersWISP at #WISPAPALOOZA 2025 by @wispanews Wireless Internet Service Providers Association:

https://www.youtube.com/watch?v=6e2MSaOQAoA

Our @herberticus and Robert shared all the latest news from our development, new features and more!

#QualityOfExperience #bufferbloat #latency #QoE #bandwidth #WISPAPALOOZA2025 #WISPA #OpenSource #RFC8290 #FQ_CoDel #sch_CAKE #broadband #QoS #ISP #WISP #FISP #FTTH #FLOSS #BandwidthIsALIE #FTTX #DaveTaht #WISPAPALOOZA25 #FWA

Yes, #LibreQoS and other #QoE middle-boxes could be a potential solution in such situations.
cc @socialhack @denog

https://www.youtube.com/watch?v=AAXwZ23YwEY

#latency #bufferbloat #QualityOfExperience #QoS #AQM #FQ_CoDel #sch_CAKE #RFC8290 #Peering #IXP #IX #InternetExchange #bandwidth #DENOG17 #DENOG
#OpenSource #BandwidthIsALIE #FLOSS #FQCoDel #schCAKE #jitter #FTTH #FTTX #ISP #broadband #FWA #FISP #WISP #NetNeutrality

Tạo Mapnitor Analytics để giám sát máy chủ! Theo dõi thời gian hoạt động, độ trễ và lịch sử ngừng hoạt động trong một bảng điều khiển sạch sẽ. Nhanh chóng, đơn giản, không phức tạp #MapnitorAnalytics #GiámSátMáyChủ #Uptime #Latency #DowntimeHistory #SaaS #PhầnMềm #CôngNghệ #GiảiPháp #TốiuHóa

https://www.reddit.com/r/SaaS/comments/1ouzcl8/i_built_this_so_id_stop_finding_out_my_servers/