It’s not just fertilizer:

it takes about 7.3 units of (primarily) fossil energy to produce one unit of food energy Assessing the sustainability of the US food system: a life cycle perspective

With all the fertilizer, heavy equipment and agricultural practices the food production today is very inefficient from an energy perspective.

Without cheap, abundant energy available the whole food production system is not sustainable