mothcomputereally scared to run tools like sort on this file
im verifying that it parses correctly by just running jq on it and its taking minutes. actual whole minutes. several of them. and this is on a very powerful server, with an nvme drive, a lot of cache, and some very fast memory
fops@mothcompute are you using jq stream mode
@chfour well i passed it a file argument so i would hope not
@mothcompute i mean stream mode is specifically made for parsing huge files so you might want to check that out
@chfour oh i see... i dont think this will be useful here though, since its in 'json lines' format
@mothcompute i think it should still work
Mattias Wadman@chfour @mothcompute i think jq --stream might mostly save on memory usage. to go faster and do jq stuff maybe try jaq? if just verify maybe something simdjson-based?