@herrbischoff As far I know and I can be wrong 😉, there is no option to have more than 16 colors in a pure TTY. That's the limitation of `vt`, according to the FreeBSD wiki and its man page. If you want more, you have to use at least framebuffer, which requires some X11 libraries. And as far I know, X server doesn't use that much memory, usually less than 100 MB.
X11 is not required in order to have a terminal emulator on a framebuffer. jfbterm, zhcon, and my console-fb-realizer all do so without X11.