Don't fear long text files

I work with text files a lot. Whenever a text file breaks the 1000-line mark, I tend to split it up into multiple files. And whenever a line gets "too long" (i.e. more than about 200 characters), I start to feel uneasy, afraid that it might break something. I get this with Gemini files a lot, because Gemini encourages you not to hard-wrap lines, but to put an entire paragraph into a single line.

I am now trying to overcome those fears. The key is to think about text files as binary files.

What is a text file?

A text file is a file that consists of zero or more lines. A line is a sequence of non-null bytes that ends in a newline byte.

File size isn't the problem

Imagine you have a text file, and you append 10 lines of about 100 bytes every day -- who you met, what you ate, where you went, what you did. That's about 1000 bytes per day. Round that up to 1 KiB.

Imagine you do that for a year. You end up with about 365 KiB per year. Round that up to 400 KiB.

Now imagine you do it for 10 years. That's about 4000 KiB total. Round that up to 4 MiB.

Ten years of ten lines every day, and you end up with less than 4 MiB in your file.

Now imagine you take a single unnecessary photo with your smartphone camera -- an accidental shot at a wrong angle, or multiple shots of the same scene (just to be safe). You immediately end up with more than 4 MiB in that file.

And if an image manipulation program can store your accidental 4 MiB smartphone photo in your RAM (and way bigger photos too), then a good text editor can easily handle your 4 MiB decennial diary as well.

Line length shouldn't be a problem either

To continue with the image file example: an image file is a binary file. It can be arbitrarily large and can contain arbitrary bytes. In order to process it, a program has to read it in byte by byte, dynamically allocating (and reallocating) the memory to store it.

If a program handles text files in the same way -- reading it in byte by byte, allocating memory dynamically, and occasionally stumbling upon newline bytes, unbothered by their frequency or infrequency -- then it can handle lines of arbitrary length.

Problems can only occur if a program tries to read an entire line at once and store it in a pre-allocated buffer of a limited size. That's why POSIX says that lines shouldn't have more bytes than LINE_MAX, which is usually 2048 (you can check yours with "getconf LINE_MAX").

But that is hardly anything to worry about. It is hard to write a line longer than 2048 bytes in the first place. But if it does happen, and if it does lead to a problem, then it is one of the first edge cases a good program will check for. It will print an error message and quit. It won't edit your file. You can still safely copy it and carry it around (perhaps onto a more apt system), thinking about it like you would when you handle a binary file.