Emma Tosch, Thought FollowerApr 13

This question is for folks who have done some kind of computing research.

Did you ever get formal training in how to do a literature review? What about informal training?

Some options, in case that lowers the barrier to entering the conversation:

Learned in a formal course
7.3%
Learned from peers
27.3%
Learned from advisor
34.5%
Other
30.9%
Poll ended at .
Show threadEmma Tosch, Thought FollowerApr 13
As a follow-up question: what platform do you do use for search?
Google Scholar
76.9%
JSTOR/EBSCOhost/via library
0%
ArXiv
7.7%
Other
15.4%
Poll ended at .
Show threadcynthiaApr 13
@etosch
for computers: acm
for everything else: library search
in general: following citation chains from any work that i'm starting from
Show threadEmma Tosch, Thought FollowerApr 14
@cxli I originally had ACM in here but could only have four options, so I replaced it with other!
Show threadEmma Tosch, Thought FollowerApr 14

@cxli For context: the #acmdl frictions make systematic reviews painful. It feels borderline unusable as a research tool and is incomplete.

#googlescholar is more complete, but the accuracy of the metadata drops off. I've found that historic searches (e.g., <1950) are mostly incorrectly dated.

I was curious whether this is corroborated by research and came across: https://pmc.ncbi.nlm.nih.gov/articles/PMC7079055/
...

Checking your browser - reCAPTCHA

Show threadEmma Tosch, Thought FollowerApr 14

@cxli Interestingly, this study (conducted in 2019) reports that the #ACMDL allows bulk download. I don't know if this feature is just hard to find or if it's been removed since then.

(Maybe @JonathanAldrich would know?)

Show threadJonathan AldrichApr 14
@etosch @cxli I don't know the history but right now I think they are doing it as a defense against unauthorized LLM training and other things that act like DDOS. It can cause problems for certain kinds of academic use; given this, I'm honestly not sure it's worth the cost.
Show threadEmma Tosch, Thought FollowerApr 14
@JonathanAldrich @cxli I've had several research threads over the past 3-4 years that have more or less stalled out because while the DL seems like the best resource for them, it's just too labor intensive to manually search, click, download, refine the search, exclude papers already read, etc.
Show threadJonathan AldrichApr 14
@etosch @cxli What is the alternative to manual search? You want to script the search? Would you use LLMs or other tools to read the downloaded papers and automatically decide if they are relevant to your systematic review?
Show threadEmma Tosch, Thought FollowerApr 14

@JonathanAldrich @cxli re:LLMs. I guess I might consider using an LLM to verify aspects of the review, but not for the primary research.

Here's an example task I recently tried to do: I wanted to catalogue the benchmarks used in ASPLOS 2026 papers. My query was very simple: just the papers from the proceedings that use the word "benchmark" somewhere. I wanted a table of the names of the suites, domain, units (or "entity types"), size, dates of introduction, and a few other things.

Show threadEmma Tosch, Thought FollowerApr 14
@JonathanAldrich @cxli The problem was that just programmatically extracting a list of the names of the papers and their DOIs doesn't seem possible without (I'm assuming) breaking the ACM DL's terms of service.
Show threadEmma Tosch, Thought FollowerApr 14
@JonathanAldrich @cxli Instead I get my paginated list of papers and I have to click through each of them, copy their names and DOIs, then open them, and then fill out my table.
Show threadJonathan AldrichApr 15
@etosch @cxli Yeah I can see why you'd want support for automation. I think it's totally reasonable, but I don't think the ACM has been thinking about these uses.
Show threadEmma Tosch, Thought FollowerApr 15
@JonathanAldrich @cxli Follow up question: do you happen to know if the ACM employs librarians or archivists?
Show threadJonathan AldrichApr 15
@etosch @cxli Not sure about staff. I know we have some on the Publications Board. If you are interested I can try to make a connection.
Show threadEmma Tosch, Thought Follower
@JonathanAldrich @cxli Yeah, actually that would be very cool, thank you!
Apr 15 at 3:53pmWeb
Show threadJonathan AldrichApr 15
@etosch @cxli Ok, reflecting on this, providing more service from the DL is in the purview of the ACM DL Board, not the Publications Board (which really does publications policy). Some members of the DL Board with specific library expertise include Stephen Downie (UIUC, Ph.D. in Lib/info sci), Michael Ley (creator of DBLP), and Phoebe Ayers (Librarian at MIT).