Cloudflare now serves sites in Markdown to AI agents

https://lemmus.org/post/20178267

A few things come to mind 1. Is this much different from the “reading view” popularized by Instapaper, reading later, etc and now baked into most browsers today? 2. What is a token? 3. How is it that tokens have become the base unit on which we denominate LLM work/effort/task difficulty/cost? 4. Does every LLM model make use of “tokens” or is it just one or a select few? 5. If multiple use the idea of tokens, is a token with one LLM/provider equivalent to a token with a different LLM/provider?

Here’s an OpenAI page that allows you to enter text and see how it gets tokenized:

platform.openai.com/tokenizer

OpenAI Platform

Explore developer resources, tutorials, API docs, and dynamic examples to get the most out of OpenAI's platform.