What is a topological space?

It is a mathematical device to define what a continuous function is, in a general setting.

1. A topological space is a set X together with a collection of subsets of X, called open, such that finite intersections of open sets are open, and arbitrary unions of open sets are open.

2. A function of topological spaces is continuous if inverse images of open sets are open.

What is the intuition behind (1) and (2)?

I claim that it is better to ask, instead, how mathematicians came up with (1) and (2).

1/

At the beginning, people considered continuous functions ℝ → ℝ.

One of the wrong intuitions, at that time, was that such a function is continuous if "you can draw it without lifting the pen".

A counter-example is Cantor's devil's staircase. This continuous function you can't draw, with or without lifting the pen.

https://en.wikipedia.org/wiki/Cantor_function

Mathematicians spoke of continuous functions for a long time before there was a precise definition of continuous function. This was a vague idea, which, nevertheless, was useful.

At some point the definition of continuity for a function f : ℝ → ℝ was elucidated.

∀ x ∈ ℝ , ∀ ε > o, ∃ δ > 0 , ∀ x' ∈ ℝ, ∣ x - x' | < δ → | f(x) - f(x') | < ε.

This definition allowed a lot of theorems to be proved rigorously. This is why it was useful.
A lot of theorems that were claimed, could now be proved.
2/

Cantor function - Wikipedia

This definition *does* have an intuition.

Suppose you want to calculate f(x₀), for example because you are an engineer and want to build a bridge, where x₀ is a physical quantity. Unfortunately, we can't measure x₀ exactly.

But you still want to know what y₀ = f (x₀) is, at least approximately.

But "approximately" doesn't mean anything. You want to know y₀, say, with two correct decimal digits. This will do to build a robust bridge.

Then the question is, how many correct digits of x₀ do you need to know, in order to get the desired two decimal digits of y₀?

More generally, we want the following to be the case. In order to know n digits of output of the function, it is enough to know m digits of the input, where m depends on the input and on n.

This is possible if and only if the function is continuous.

So one intuition about continuity is that "finite amounts of output depend on only finite amounts of input".

3/

But this was for functions ℝ → ℝ only.

Soon people started to consider continuous functions "of two variables", that is, functions ℝ × ℝ → ℝ, and then of n variable, so ℝⁿ → ℝ, and in each case a different definition was needed.

Moreover, because people were trying to e.g. solve differential equations, which amounts to given one *input* function (called the initial condition) to figure out an *output* function (the solution), people came across functions mapping continuous functions to continuous functions, which, themselves, may or may not be continuous.

So a general definition of continuity was needed to clean up the mess and be able to make progress more efficiently.

A first, rather useful, general setting was that of a metric space. You say what the distance between two things (e.g. real numbers, tuples of real numbers, continuous functions) is. The axioms for metric spaces are very intuitive, and I won't reproduce their statement here.

But I want to say this.

1. The definition of continuity is the same as above, with the absolute value of the difference replaced by the distance function d. A function f : X → Y is continuous iff

∀ x ∈ X , ∀ ε > o, ∃ δ > 0 , ∀ x' ∈ X, d(x,x') < δ → d(f(x),f(x')) < ε.

2. We can then define a set U to be open if for every x ∈ U there is ε > 0 such that every x' with d(x,x') < ε is in U.

3. The open sets are closed under finite intersections and arbitrary unions.

4. A function is continuous in the above ε-δ sense iff inverse images of open sets are open.

4/

This, and much more, is beautifully explained in the book

George F. Simmons. Topology and modern analysis
https://archive.org/details/introduction-to-topology-and-modern-analysis-simmons

which I highly recommend.

What happens next is that metric spaces are not enough. There are sets on which we want to consider continuous functions which can't be metrized so that metric continuity coincides with the notion of continuity we want.

I like this book because it starts from trivial things, making them less and less trivial as we progress, until it gets eventually to many things, including Stone duality. I self-learned topology from this book as an undergrad.

5/

Introduction to topology and modern analysis : Simmons, George Finlay, 1925- : Free Download, Borrow, and Streaming : Internet Archive

This is a classical introductory textbook on topology and analysis

Internet Archive
@MartinEscardo : But open-set spaces aren't enough either; for example, convergence in measure can't be defined by open sets. (Of course, this is convergence, not continuity, but they're interdefinable, and you could tell an equivalent history based on convergence instead of continuity. In any case, I'm sure that somebody has considered maps between function spaces that preserve convergence in measure, in other words continuous maps between such spaces, but I'm no measure theorist, so I don't know what they are.)
@MartinEscardo : (This is also a different sense of not enough, one closer to the origins of the topic in analysis, than the sense that leads to locales and toposes etc.)