I now have almost four years of electricity and natural gas use for my house via our utility companies. We put in a heat pump in May 2022. Previously, we had two window ACs in bedrooms that we ran almost entirely at night (and only some nights) and a bunch of fans. Last summer, with intense heat, you can barely see an increase in usage—particularly when you factor in inflation and a slow climb upwards in rates. (These are all two-month billing periods and average temps.)
This winter, with some bitter cold temps for Seattle last month, it looks like we paid about 5% less with the heat pump than the same period last year. But it's likely we used substantially more therms to heat the house. (Still compiling that data; it's hard to get apples-to-apples.) This chart shows hours used for AC, heat pump heat, and nat-gas heat. (We're still using the nat-gas furnace a bit; need to consider whether turning it off turns into a maintenance problem.)

@glennf Should you stop using the gas furnace completely, the utility may take that as a signal to shut off your line after some months. You may only discover this years later, during a stretch of record cold temperatures.

This is the voice of experience.

I would clear this with your gas utility, or just run the furnace 2-3x a year to make sure the gas supply is there, unless you are sure you'll never need it (newer heat pumps have solid low-temperature performance).

@hydrocabron We've still got a tankless gas water heater. But that is great to know!