Eugen Rochko
So after joinmastodon.org has received so many improvements... How do we test how well it works? 🤔
Sep 16, 2017 at 11:53pmWeb
Show threadSydney Falk, An Oversensitive SJW TypeSep 16, 2017

@Gargron

Informal method one: Find someone who's never been there that you know, show them the new one, have them try to make a new account, watch where they get hung up.

Formal method one: Set up those "alternative" trial thingies I forget the name of, where one version of something is served to people following the link sometimes, and the other is shown the other times? I think that's how that works? And then look at time-before-leave-page and time-before-trying-to-sign-up, etc.

Show threadEugen RochkoSep 16, 2017
@sydneyfalk that's an a/b test and it requires analytics, of which there are none on the page! i'm not sure if people would look down on me if i added tracking to that page, lol
Show threadSydney Falk, An Oversensitive SJW TypeSep 16, 2017

@Gargron

well, as with all things, there are an infinity of 'third options' -- you could formally track informally having friends use it and examine their verbal feedback, ask them specifically for "what was hard to find/figure out", etc.

it's HARD to ask the right questions of humans, s'why analytics is blowing up so hard, at least in large part

but if you keep track and make sure you're asking simple, basic questions of testers, you'll get an idea of how it's going, that might be enough.

Show threadSydney Falk, An Oversensitive SJW TypeSep 17, 2017

@Gargron

Of course, you could NOTE that you're adding tracking, temporarily, to try to make the page more usable.

Honestly -- and I know this will sound cynical, but it's the truth, AFAICT -- but no matter WHAT changes, some people always complain, no matter what. You literally cannot do anything without upsetting some nonzero number of users.

I know that might seem like advocating inaction or apathy, but it's meant more as a reminder that humans overall deal badly with change.

Show threadSydney Falk, An Oversensitive SJW TypeSep 17, 2017

@Gargron

Stress scales used in psychology include "negative" events (like divorce) at change levels similar to "positive" events (like weddings). Stress, in some ways, isn't about improvement or worsening. Some of it's just about the change itself.

People vent a lot, or they'll have a bad day and piss and moan for a week about one button getting moved somewhere that helps fifty other people. Forty-nine of the fifty might say fuck all about the improvement, too.

Show threadSydney Falk, An Oversensitive SJW TypeSep 16, 2017

@Gargron

(but I will also say: the fact you're ASKING yourself that question puts you a little ahead of the game at all, b/c a lot of developers seem to see human use stuff as "well, we can't REALLY test this, blah, whatever, let's just see who complains")

Show threadLeigh GarlandSep 17, 2017
@Gargron what was the impetus behind the changes in the first place?
Show threadEugen RochkoSep 17, 2017
@toychicken Negative feedback from personally asked people
Show threadLeigh GarlandSep 17, 2017
@Gargron okay that's a valid start, strong opinions often uncover useful directions. However, no-one comes without bias :)
What's your #1 objective? Grow Mastodon? Get more users? Improve features?
Whatever that priority is, you need to baseline your existing work before making further changes. Then, as you make changes, see what impact they make.
You're only a small team, so it'll be hard to track more than one or two effects at once, so be judicious about what you want to track.
Finally...
Show threadLeigh GarlandSep 17, 2017
@Gargron ... It's likely that your message on joinmastodon is already pretty good, and that to increase numbers, you'll need to reach people beyond your network. That means marketing.
Show threadEugen RochkoSep 17, 2017

@toychicken Grow Mastodon/Get more users (same thing) is joinmastodon.org's primary objective

Improve features is Mastodon's primary objective

Show threadLeigh GarlandSep 19, 2017
@Gargron in that case, your challenge is all about the marketing. Initially, it's a volume game. I usually assume a 1000:1 ratio. Get 1000 people to see something, 1 might sign up.
Over time, you'll see where your rich seams are and you can focus on those to get more for your money.
You've had good press in the past, so it's time to hit up any friendly folk that showed interest in the past and let them know what's changed since they last looked, and remind them why Mastodon is so important.