wil bierbaumFeb 27
Michael Stapelberg 🐧🐹😺

They always say computers are so good with numbers, but you can’t even store 2000 numbers in a single file!!

Try it out for yourself:

seq 1 2000 | tr '\n' ' '
cat > /tmp/wat.txt
<copy&paste the 2000 numbers from above, press enter, then ctrl+d to end input>
cat /tmp/wat.txt
wc /tmp/wat.txt

Or see my screencast here: https://asciinema.org/a/fLhMt640MqFWGuug

On a Mac, I can save even fewer numbers! But at least I get feedback that the line is full. How many numbers can your computer save?

Feb 26 at 8:30pmWeb
Show threadMichael Stapelberg 🐧🐹😺Feb 26
Tested with URxvt, GNOME console and foot on Linux, Terminal.app and Ghostty on Mac.
Show threadcambid83 (he/his)Feb 26
@zekjur can confirm this for kitty too.
Show threadMatt MascarenhasFeb 26
@zekjur Interesting! That 4096 is a suspiciously round number. Wonder if this is telling us that stdin works with a page size of 4K, and `cat > $filename` somehow only reads one of the pages?
Show threadscyFeb 26

@zekjur Dafuq.

Interestingly enough, all the numbers are in the clipboard, pasting them in Neovim works fine. Pasting them directly into the shell command line works too. It's just the input redirection somehow?

It's not cat either, dd does the same thing 🤔

And yeah, 4096 bytes (including the newline), just like you. (Gnome Terminal, bash, Wayland)

Show threadRainer MüllerFeb 26

@scy @zekjur It's the expected behavior of a tty on Linux in canonical mode:

From termios(3):

> The maximum line length is 4096 chars (including the terminating newline character); lines longer than 4096 chars are truncated.

Show threadRainer MüllerFeb 26

@scy @zekjur To prove it, you can disable the canonical mode for the tty with `stty -icanon` before pasting (and restore the previous tty settings afterwards). The canonical mode is what allows you to edit the input line by handling special characters such as backspace (erase), Ctrl-W (werase), or Ctrl-D (eof).

stty_saved=$(stty -g)
stty -icanon
cat > /tmp/wat.txt
<copy&paste the 2000 numbers, hit return, then kill it with Ctrl-C>
stty "$stty_saved"
wc /tmp/wat.txt

Show threadChristian KuglerFeb 27
@raimue @scy @zekjur If you use <<EOF or <<"EOF" it also works fine. Do you know why it works in this case? Maybe stdin in controlled by the shell instead of the stty or something? I always assumed "EOF" and <ctrl-d> would be identical
Show threadRainer MüllerFeb 27

@syphdias @scy @zekjur With the heredoc syntax, input will be processed by readline in the shell. That also means the whole input is buffered by the shell before the process for the `cat` command is even forked. Your shell will create a temporary file and redirect that to stdin instead of reading directly from the tty.

You can check the tty settings from a different terminal window to observe this. In the first window, get the pty device with the `tty` command. In the second terminal window, check with `stty -a -F /dev/pts/XX` while the first terminal window is in the shell prompt, running cat, running cat with heredoc, etc. Whenever readline is processing input, the tty will be in non-canonical input mode indicated by '-icanon'.

Show threadChristian KuglerFeb 27
@raimue @scy @zekjur In my case I didn’t use cat with heredoc but just the shell (zsh) itself. So heredoc is closer to <file or piping output directly from earlier command (unnamed pipe).
And in this case the terminal mode actually changes to write to the pipe first (you said file, but I assume it is a pipe). Did I get that right?
Show threadRainer MüllerFeb 27

@syphdias Right, reading the heredoc is fully handled by the shell. For bash that would be using readline, for zsh it is zle. The shell writes the input to a temporary file [1] and therefore passing that as input to a command works just like any other redirection.

The important difference is that the shell uses the tty in non-canonical mode and handles the editing itself. The shell changes the tty settings for the prompt and restores them (to canonical mode) when a process runs that is directly attached to the tty. Therefore, when the shell is reading the input, is not the kernel tty driver that has to handle special characters for editing, which would have the line length limit of 4096 characters.

[1] As I looked it up now, bash >= 5.1 actually has an optimization to use a pipe(2) if the heredoc input fits in there without blocking. Otherwise it falls back to the temporary file. I assume zsh might do something similar. https://cgit.git.savannah.gnu.org/cgit/bash.git/tree/redir.c?h=bash-5.3#n419

redir.c - bash.git - bash

Show threadChristian KuglerFeb 27
@raimue thank you for the insights!
Probably not interesting to you but people who might read this. There is a great article about tty which also talks about how modes. https://www.linusakesson.net/programming/tty/
The TTY demystified

Show threadZeljko Stevanovic 🐧Feb 27
@zekjur buffer limit on tty in cononical mode. more on https://unix.stackexchange.com/questions/131105/how-to-read-over-4k-input-without-new-lines-on-a-terminal
How to read over 4k input without new lines on a terminal?

So I have a lot of data WITHOUT NEW LINES on the clipboard (it's a large SVG file on one line). I went $ cat >file.svg then tried to paste (in Gnome Terminal), but only the first 4kB characters...

Unix & Linux Stack Exchange
Show threadZeljko Stevanovic 🐧Feb 27

@zekjur cat in non canonical mode. ^d finish program

https://gist.github.com/zsteva/4bf1f6088d7b91de6ee76ab1eb0fa396

stdio workaround for tty buffer line limit.

stdio workaround for tty buffer line limit. GitHub Gist: instantly share code, notes, and snippets.

Gist