All three of these involve walking through a sequence of steps:
bytes -> characters -> rendering -> reading
...with all but the last step occurring in the computers.
In the case of emoticons, the bytes become pictures at the reading stage. : does not represent eyes and is rendered as a colon character, - does not represent a nose and is rendered as a hyphen-minus character, and ) does not represent a mouth and is rendered as a close-parenthesis characer ... but in the eyes of the reader, :-) registers clearly as a smiling face.
In the case of smileys, the bytes become pictures at the rendering stage. Again, none of the *characters* represent pictures, but the computer recognizes a sequence of characters (e.g. :geek: on a phpBB forum) and renders this sequence of characters as an image.
In the case of emoji, the bytes become pictures at the *character* stage. 😸 represents *as a character* a picture of a "Grinning Cat Face With Smiling Eyes", on top of being rendered as a picture (😸) and perceived as a picture by the reader.
...that said, "emoji" as a term has undergone an expansion in meaning, probably through the influence of group chat services like Slack and Discord, and now includes smileys. But that is how I think of the distinction between these three things.
*sets chalk down* Any questions?