Ah, forever uncertain how our seniors see us.
First, he "... used me with great respect, and discoursed upon his business as with one that he did esteem of", which we like very much.
Then, "by and by he went away, forgetting to take leave of me, my back being turned, ...". Ouch, this hurts. Must we now doubt the respect and esteem we thought we saw earlier?
It's tough to rise through the hierarchy through hard work and diligent attention to the wishes of our superiors.
And why was our back turned and we weren't paying complete attention to our senior? We were "looking upon the aviary, which is there very pretty, and the birds begin to sing well this spring." Oh, Sam, you are a good man. The beauty of the bird song asserting its primacy over your attention.