@andrew @smach this I think is the key observation: students learning the stuff don't have enough baseline knowledge to understand whether the stuff coming back from chatgpt (or google or chegg) is any good.
I make a point that everything in my courses can be done using the course materials, with the exception of a few places where I explicitly ask students to look up something and report back on what they find (with a citation to where they found what they were looking for).
This might interest you.
I wrote it to basically plead with my university to get off the bandwagon.
They didn't really listen: you cannot convince people to understand AI when their salary depends on the AI hype