Mastodawn

Bug Mar 20, 2024

Several of us overly online biologists spent years quietly doing an experiment on Twitter, trying to find out if tweeting about new studies from a set of mid-range journals caused an increase in later citations, compared to set of untweeted control articles.

Turns out we had no noticeable effect; the tweeted papers were cited at the same rate as the control set.

Our paper, headed by Trevor Branch, was published today in PLOS One:

#SciComm #Twitter #X #Science

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0292201

Controlled experiment finds no detectable citation bump from Twitter promotion

Multiple studies across a variety of scientific disciplines have shown that the number of times that a paper is shared on Twitter (now called X) is correlated with the number of citations that paper receives. However, these studies were not designed to answer whether tweeting about scientific papers causes an increase in citations, or whether they were simply highlighting that some papers have higher relevance, importance or quality and are therefore both tweeted about more and cited more. The authors of this study are leading science communicators on Twitter from several life science disciplines, with substantially higher follower counts than the average scientist, making us uniquely placed to address this question. We conducted a three-year-long controlled experiment, randomly selecting five articles published in the same month and journal, and randomly tweeting one while retaining the others as controls. This process was repeated for 10 articles from each of 11 journals, recording Altmetric scores, number of tweets, and citation counts before and after tweeting. Randomization tests revealed that tweeted articles were downloaded 2.6–3.9 times more often than controls immediately after tweeting, and retained significantly higher Altmetric scores (+81%) and number of tweets (+105%) three years after tweeting. However, while some tweeted papers were cited more than their respective control papers published in the same journal and month, the overall increase in citation counts after three years (+7% for Web of Science and +12% for Google Scholar) was not statistically significant (p > 0.15). Therefore while discussing science on social media has many professional and societal benefits (and has been a lot of fun), increasing the citation rate of a scientist’s papers is likely not among them.

Alex Wild Mar 20, 2024

We were able to increase the amount of views and downloads those papers got, though. We could get eyeballs on the science.

I guess you can lead a horse to paper, but you can't make him cite.

Alex Wild Mar 20, 2024

The good news is, if you thought Twitter's descent into Musk-filled madness might be detrimental to your efforts to get other scientists to cite your work, fear not. In this regard, Twitter was not actually that useful.

Alex Wild Mar 20, 2024

There have been several studies showing that the highly-tweeted papers are also highly-cited.

I think that's right. But not because tweeting causes citations.

In light of our results, it seems more likely that both social media communicators, and publishing scientists, recognize impactful work when they see it. Good science just gets talked about more, regardless of the medium.

That scientific research impact can't easily be gamed by social media I find quite reassuring.

Nathan Lowell (he/him) 📎Mar 21, 2024

The problem with social media is that it's soylent green all the way down.

Janne Moren Mar 20, 2024

@alexwild
So, people who would cite your paper will usually find it without any social media self-promotion on your part. That'll be a relief for quite a few people.

David Benfell, Ph.D. (he/him/his)Mar 21, 2024

@jannem @alexwild I would expect most scholars to perform their literature searches using databases at academic libraries. This work doesn't confirm that but it's what students are taught to do.

Ron Parsons Mar 20, 2024

@alexwild
I use social media to see interesting papers outside of my usual reading. So probably accessing things I may never cite in a formal paper but helps in other ways (teaching, general knowledge, etc)

franco_vazza Mar 22, 2024

@alexwild 😅
all that time I lost on X!!

Starraven Mar 20, 2024

@alexwild
While worthwhile for the evidential data, I really would never have considered that people who would need to cite a scientific article would be looking for them on Twitter.

Dan Goodman Mar 20, 2024

@alexwild brilliant work, thank you for doing this experiment! I'm wondering if you might expect a different result for tweets of your own work rather than someone else's? The reasoning being that your followers are more likely to be the sort of people that might cite your work in future than the followers of a general science communicator. I would for sure argue that for most people the majority of their followers are unlikely to cite, but for your own work you might be reaching the critical audience. One of the reasons I'm on twitter as a follower rather than tweeter is that I see papers from people in my extended community that I wouldn't have seen in the journal that it got published in. It would seem surprising if this effect was totally negligible given that twitter is the source of a substantial fraction of the papers I read. (Although less recently, the quality of twitter really has noticeably declined of late.)

Steve Gisselbrecht Mar 21, 2024

@alexwild
This is wonderful. Everything about it.

Scott Matter Mar 21, 2024

Nice one! Really cool study and interesting result.

It’s helpful to think about where and how to focus my efforts in relation to the kind of metrics my institution will recognize and value.

Tangent: any recommendations on studies that look at other forms of impact (ie harder to measure things like collaboration, non-publication outcomes, etc)?

Cameron Neylon Mar 21, 2024

Nice work! This takes me back to speculative musings on the time domain behaviour of these interventions (http://hdl.handle.net/20.500.11937/32897, your Figure 2 made me think of my Figure 4)

You've really captured the immediacy of the viewing effect and I'm wondering whether a citation effect might be clearer if analysed in a more time dependent way rather than at a three year census point...

...but you've given us the necessary information to make that analysis possible, which is fabulous! (whether I have the time is another question)

The other question I've got is whether the citations might show greater diversity (reaching a wider range of scholars) because they are coming through a set of followers that might have wider geographic or disciplinary diversity. And we can test that as well! (same caveats apply...)

The road less travelled: optimising for the unknown and unexpected impacts of research

John Towse Mar 21, 2024

@alexwild @cameronneylon
Yes, this makes sense…
There are direct citations (citing X because I’m replicating X / extending X by taking the next step / assimilating X into a theory) and there are more indirect citations (citing X because it’s interesting & cool & maybe it can link to these data Y). Social media might be expected to pull more of the latter, but evidently not noticeably so. A deeper dive into the non-sig citation gain might examine this diversity

Cameron Neylon Mar 21, 2024

@johnntowse @alexwild The other point is that using a bigger citation data source might give a different result if there is a real effect but the effect size isn't huge and the statistical power not quite there. That's another thing that would be relatively easy to test with OpenCitations and the DOIs (I'll put it on the list...)

John Towse Mar 21, 2024

@cameronneylon @alexwild
The issue of power is discussed in the paper of course, but I am sympathetic to the argument there the effect size is not that impactful (even if it exists at the population level, it’s not making much difference for the individuals who do or don’t tweet about their papers)

Cameron Neylon Mar 21, 2024

@johnntowse @alexwild

Agreed, my counter would be that in many of these cases the distribution of effects amongst individual outputs is wild, so effect sizes may look small on average but the effect when it happens can be quite large. And I would always have expected any effect to be large, but for a subset of papers.

Obviously randomised control trials like this to smear some of those effects out by design.

I feel that a Hidden Markov Model or time domain analysis would ultimately help in understanding the underlying pathways. But I also get that those approaches tell us about probabilistic associations, not causality - which is where the approach here is strong

And all of that said your main point is well supported - that for any specific paper, being tweeted about doesn't (didn't?) lead to significantly more citations on average

John Towse Mar 21, 2024

@alexwild @cameronneylon
Absolutely, these are really interesting questions to think about in response to a clever paper. (And in the meantime those who stay away from social media / certain social media can modulate their FOMO!)

llewelly Mar 21, 2024

@alexwild
interesting. I think while on twitter I followed about 8 of the 11 authors, maybe more. (But, I am not a scientist of any kind.)

John Towse Mar 21, 2024

@alexwild Nice read, thanks!
I’ve sometimes wondered about the value of the research conference circuit in getting others to know about work (an offline analogy to this), thinking about the availability heuristic (if you can recall something more easily it factors into your decision making). However, maybe the degree of engagement is different anyway (listening to a talk vs the time taken to retweet) ?

Furqan Shah Mar 21, 2024

@alexwild The elephant in the room is "if there are -big name- co-authors on a paper or not". If all the authors are relatively obscure/lesser-known names in their respective field(s), nobody is going to take their science seriously (no matter how groundbreaking or robust). And if so, these papers will get cited on merit, *if* they really must be cited, and *if* someone has been able to reproduce the findings. With information overload, the value of individual papers tanks substantially. Sorry!

Christopher Kyba Mar 21, 2024

@alexwild It's a neat experiment, but the presentation suffers from a classic problem: the results are consistent with an effect, but it's presented as if there is none, because it's not "statistically significant".

Since the CI overlaps zero with a mean 12% citation increase, it likely also overlaps 25% increase in citations. "Tweeting has no noticeable effect" and "Our results are consistent with tweeted papers having 1/4 more citations" are wildly different presentations of the same result.

Peter Kraus Mar 22, 2024

@skyglowberlin @alexwild I'm not a statistician, but surely "papers will be cited 25% more" and "papers will be cited 0% more" are two different hypotheses, and only one of them makes sense as a null known upfront.

Christopher Kyba Mar 22, 2024

@pkraus @alexwild Being unable to reject the null hypothesis is often presented as evidence of no effect, when in fact it's just evidence that the effect size is most likely below some upper bound - in this case, probably somewhere around a 25% increase.

If the final confidence interval had been [1.01, 1.26], it's likely the presentation would have been "wow, twitter increases the citation rate" even though the result would have been nearly the same as what these authors presented.

Alex Wild Mar 22, 2024

@skyglowberlin @pkraus Yes, this is a good point. We estimated we’d need an experiment several times the size of the one we ran, given the effect size.

From the standpoint of an individual researcher, I’m not sure a very slight statistical increase in citations over a large sample would change the underlying recommendation that it’s better to increase citations by writing good papers than to attempt to boost citations on social media.

Christopher Kyba Mar 22, 2024

@pkraus @alexwild This paper is in some sense just for fun, so it doesn't really have real world consequences. But the same interpretation problem happens with research that does affect all of us. Here's an example that my colleague Thomas Kantermann and I critiqued several years ago: https://pure.rug.nl/ws/portalfiles/portal/239811305/Hormones_2016_142.pdf

To put it another way: it is never possible to confirm a null hypothesis. The most one can do is limit the possible effect size.

Peter Kraus Mar 24, 2024

@skyglowberlin @alexwild again not a statistician, but isn't the whole point of a t-test to reject a null hypothesis? Choosing your upper bound, like you're doing here with the 25%, after you've done the analysis, cannot possibly be right.

Of course, the t-test tells us that there's a 15% probability all of the observed higher citation numbers are "by chance", which means there's 85% chance they're not, which might be good enough odds for some to keep tooting about their papers.

Christopher Kyba Mar 24, 2024

@pkraus @alexwild The opposite of "rejecting the null hypothesis" is not "confirming the null hypothesis", it is "finding consistency with the null hypothesis". It is not possible to confirm the null hypothesis, only to find that effect sizes are so small that you don't care.

A statistical test doesn't tell you the probability that an observed mean value occurred by chance, it tells you that if there was truly no effect, then you would expect to observe a mean value that large X% of the time.

Craig Aaen Stockdale Mar 21, 2024

@alexwild Thank you for doing this! What about other types of impact? Citations in policy docs, for instance?

Tofu Golem Mar 21, 2024

@alexwild
That's a relief.

levampyre Mar 21, 2024

@alexwild I wonder, if it were the same for posting on Mastodon. My feeling is, that there are more science interested folks around here.

Robert Dale Parker Mar 21, 2024

Fascinating: thank you.

This question may reflect my humanities perspective (or maybe not—I don't know), but if the tweeting did not influence the number of citations, is there any possibility that it influenced the kind of citations?

Joshua Drew Mar 21, 2024

@alexwild Mastodon high five (hoof?)

j_bertolotti Mar 22, 2024

@alexwild On the other hand, I would never have seen this paper if a person I follow didn't boost your post 😉
That said, I am unlikely to cite it, because my papers are on a totally different topic 🤷🏻‍♂️

Bruce_Ak Mar 22, 2024

@alexwild No link found between sense, productivity and Twitter

Who knew ?

Dr. Jürgen Knödlseder Mar 22, 2024

@alexwild @Luke_Drury Great work. Now let’s repeat it for #mastodon 😅

Chiasm Mar 24, 2024

@alexwild Cool, good to know--love the experimental design, that was really carefully done!!

Max_evolbio Apr 2, 2024