I've read through this paper a couple times and started looking at its data.
https://www.nature.com/articles/s41586-023-06767-1
I'm predicting it now before I get further into reviewing this: this paper is wrong for all the reasons you'd suspect and isn't going to hold up to any scrutiny.
But managers and non-scientists everywhere are going to use it to reign in remote workers and say that it's "supported by the science".
After reviewing "Remote collaboration fuses fewer breakthrough ideas" more thoroughly, I believe it should be retracted. Here is a write-up of why this is the case:
I can summarize the problems with this paper as follows:
1. Confounded relationships between the explanatory and response variables that are not properly adjusted for,
2. Ill-defined response variable that is insufficiently explored and whose quantitative value does not match the interpretation the authors suggest,
3. Large amounts of data with NaNs that are not discussed,
4. A large dataset with most of its variance contained in < 1% of the overall data,
...
Like I say in the blog post, the most likely explanation for the observations the authors make it Berkson's paradox:
https://en.wikipedia.org/wiki/Berkson's_paradox
The authors do not adjust for the web of confounding relationships that exist in their data, the most important relating to publication year, time since publication, and the COVID-19 pandemic.
Now I feel some sense of professional obligation to submit a "letter of concern" to the journal, but I can't find any links to do this directly.
It seems like I'm supposed to email Nature's main address to kick start the process. Does anyone have a better idea about how to do this?
Alright, I've made minor corrections to my blog post, emailed the authors expressing my concerns about this paper, and posted the concerns on @PubPeer (the comment should show up after moderator approval).
https://pubpeer.com/publications/7DD790DB75C7B6DEEDA47125241E3C
I also put my code on GitHub for easier inspection: https://github.com/jrhawley/linRemoteCollaborationFuses2023/
Let's see how this goes.
I haven't received any reply from the corresponding authors since I contacted them 2 weeks ago. I'm emailing them again to see if we can discuss these concerns.
I don't want to write to the journal unless I have to, because I know how painful that process is going to be and how much extra work it's going to take. But is that my only option if I can't get a hold of any of the authors?
@jrhawley When I was involved in something like this back in 2009, we submitted it as a "brief communication arising", a category that I think is now "matters arising":
@bstacey That was the tip I needed, thank you! I found the page:
https://www.nature.com/nature/editorial-policies/correction-and-retraction-policy
@jrhawley thanks so much for this!! I think funding is an underdiscussed confounding factor.
IMHO: Funding is highly concentrated to a few rich universities which produce some great research, while poorer academics need to take any collaborator they can get leading to more geographic dispersion.
@PhilippBayer Right, me too. I don't study productivity and the "economics of science", but from all the PIs I've talked to about running their labs, funding is the thing that PIs spend most of their time working on because it so drastically impacts the type of research they are able to do in their labs.
How can that not affect the types of papers you can publish and how widely they'll be read/cited? How is that not an important factor in how local/remote your projects and collaborations are?
@StatGenDan @bstacey Yeah, I read that and opted to contact the authors directly, first. I haven't reached out to the journal, yet. Maybe I won't have to.
I obviously need to give the authors some time to read and think about what I've written. And that Matters Arising page says Nature will consider post-publication peer review for 18 months after publication. So I don't think there's any rush for that, yet (aside from the thousands of people accessing the article while I wait, of course)
@jrhawley your point about cultural and language barriers was one of the best points in your critique.
My reading of your critique leads me to think that having an experienced humanities researcher in the team (ie. Increased diversity) would have helped.
What also would have helped would have been discussing it with researchers who are highly marginalised who are forced to do remote collaborations.
@rowlandm Completely agree. Involving domain area experts from the beginning would eliminate so many of these problems.
And you're right, the unequal impact of a paper like this on marginalized communities is so important. It is way too easy to see bad logic extending from this:
1. We need in-person work to be disruptive
2. Marginalized people need remote work to contribute fully to the workplace
3. Thus, we cannot have disabled and marginalized people on our team if we want to be disruptive
@rowlandm Is remote work perfect? No. Is it great for everyone? Also no. Are there major benefits to remote work? Yes! Let's figure out what and how they work.
I'm probably preaching to the choir, here, but just like curb cuts, remote work has huge benefits for disabled and non-disabled people, alike.
- live-transcription of talks for deaf people
- stable working conditions for new parents
- the ability to avoid bad weather and risky travel conditions
- reduced carbon emissions from commuting
@jrhawley and even the inverse of those questions give the same answer.
Is onsite work perfect? No. Is it great for everyone? Also no. Etc etc
@rowlandm Absolutely! It's always easier to question the new stuff than ask the same questions of the old stuff. As you said, in-person work doesn't work for so many people, and they've mostly been ignored until recently.
I don't like the idea that this paper can (and probably will) used to continue justifying the status quo when it doesn't show anything close to what it says it does.
@jrhawley very interesting, but you lost me at "OLS assumes the response data comes from a normal distribution." Not at all. The hypothesis tests on coefficients are only valid if the _residuals_ are normally distributed.
That said, I do agree with much of your critique, and plan to add this paper to the class on "misleading statistics".
@calderds Hey Calder, thank you! Nice to hear from you again.
Hahah, understanding linear models is important! But that is probably one of the least problematic things about this paper. The causal graphs, the confounding, the proxy variables and relationships - that's where the meat is.
If I can continue being your biostats TA for a bit longer, go watch this lecture series:
https://www.youtube.com/watch?v=FdnMWdICdRs&list=PLDcUM9US4XdPz-KxHM4XHt7uUVGWWVSus
And go read this book: http://bayes.cs.ucla.edu/BOOK-2K/
James Hawley, PhD
chiasm
Blake C. Stacey
Philipp Bayer
Daniel E. Weeks
pol_eco_pub
RM
Lars Marowsky-Brée 😷
Joris Meys
Calder Sheagren
Peter Guhl
Johannes