Seeking help! No matter how many keywords/names u search related to this story the article doesn't appear on Google
This was a front page A1 story I wrote for WaPo on how smear campaigns and abuse women journalists endure are a press freedom issue. Can someone explain why the article does not appear on Google? https://www.washingtonpost.com/investigations/2023/02/14/women-journalists-global-violence/
@leftyknowitall
It's not in WaPo's results for the primary subject but other pages are found, including an unrelated story from the same day that links to it as a "Lunchtime read."
https://www.washingtonpost.com/search/?query=Gharidah+Farooqi
Only the video included in the story is found for Juliana Dal Piva:
https://www.washingtonpost.com/search/?query=Juliana+Dal+Piva
The story is included in their sitemaps, used by search engines to index sites.
https://www.washingtonpost.com/sitemaps/sitemap-2023-02.xml
There are zero results for Zartaj Gul Wazir:
https://www.washingtonpost.com/search/?query=Zartaj+Gul+Wazir
@taylorlorenz Even searching for the URL directly in Google won't show it. Links to posts that link to the article.
The only reason I can think is this feature somehow made it on their suppression list. This is just wrong.
@fromjason @taylorlorenz Also “noarchive.” It’s been specifically barred from the Internet Archive.
EDIT: This may not be a unique thing for WaPo stories, that said
@fromjason @taylorlorenz Based on the dates in the Wayback Machine, the blacklisting happened sometime before October 2023; the last archived date for the story was June 2023.
EDIT: That may be a widespread thing on the site though. I found other stories with “noarchive.”
@fromjason @taylorlorenz Or they crawled it before the noindex tag was added.
Typically you do not want Google News to index and rank sponsored (paid for) content in the news results. But sites like USA Today, The Washington Post and many others simply cannot get their sponsor
It's difficult to reconcile "democracy dies in darkness" with "content=noindex, content=noarchive".
@mhoye @taylorlorenz maybe the real democracy was the articles we scrubbed along the way
JK I don't know why they did this. Perhaps it was an act of God. Or maybe the index grinch.
@taylorlorenz The article contains the tag
<meta name="robots" content="noindex"/>
(view source and search for "noindex"), which prevents indexing by any search engine (e.g., https://developers.google.com/search/docs/crawling-indexing/block-indexing).
It would be interesting to know when and why WaPo added this tag.
@taylorlorenz Does wWPo’s seo suck? (Probabaly, I don’t know) Any way to get them to look into it?
Too bad google bombing doesn’t work anymore. We could mount a linking campaign.
@taylorlorenz not sure this is what may be playing a role, but WaPo’s robots.txt file (one way webmasters can request robots handle things on their domain) has this:
User-agent: Google-Extended
Disallow: /
That is requesting any machine that makes a request with the user-agent set to “Google-Extended” not access (and therefore not index) anything at all on that domain.
Odd thing to have in a robots.txt file, but it is there at https://www.washingtonpost.com/robots.txt
@davidaugust @taylorlorenz Google extended is the ai search crawler, normal search still uses Google it which is allowed by the robots file
Google however has no index of the URL, which could be done by either Google in the background, or by wapo themselves in the search console, and with Google making it very difficult to submit URLs that you don't own, you would probably have to ask wapo to submit it for crawling manually to try and fix it
@taylorlorenz I found it in duckduckgo, which is Bing with a better tailor. But it's buried without adding "Washington Post" to the search. It popped right up when I did.
I didn't find it on Alexandra search, but they don't seem to include Washington Post articles. They have some bizarre sources I just noticed. Newt Gingrich 360? Wtaf?
Google's algorithms must not like you.
Maybe the fediverse needs a search engine.
@taylorlorenz
Here we have soe historic alleged #censorship
by #google.
@taylorlorenz It's marked as "noindex".
You can look up the source code.
<meta name="robots" content="noindex"/><meta name="robots" content="noarchive, max-image-preview:large"/>
Does Wapo routinely use no index?
Attached: 1 image Amazon is flooding striking workers at DBK4 in Queens with freezing water in sub-zero weather, endangering everyone. If you can, please do not shop at Amazon right now. Social media posts confirming the flooding: https://www.instagram.com/reel/DD2nNx1Sl6i/ https://www.instagram.com/reel/DD2oNicJ96-/ https://www.facebook.com/share/r/bjYgQLcLVv9NVBRn/ https://twitter.com/nycdsa/status/1870569037194989688 #UnionStrong #union #amazon #strike #law
@taylorlorenz seems like it's not in the index, but you have to be the owner of the site to use the tool to inspect the index I think.
@taylorlorenz
if you do a site search on Feb 23 some articles don't have the "noindex" tag.
BC253
Taylor Lorenz
Emelia 👸🏻
Jeremy le fou
Curtis Wilcox
KeithC
Cat 🐈🥗 (D.Burch) 
Bruce Davie
Nicholas Scott-Mills
Isocat
Quinn Comendant
MegatronicThronBanks
Bruce_Ak
✅ Evert Albers
Alan Langford 🇨🇦🧤🧊摏
Lyle Solla-Yates
Grant 🇺🇦ArmUkraine🇺🇦
Mr. Lance E Sloan (IRL) 👤
kwayk42
John Breen
Stephen Bannasch (316 ppm)
Jay Baltz
fromjason.xyz ❤️ 💻 ✍️ 🥐 🇵🇷
Ernie Smith
Matthew Malthouse
Winfried is Anti Fascism 🏳️🌈
Dan Hatton
Cassandrich
Philipp 
Scott J. 🇺🇦🇺🇸🇺🇳
mhoye
Jed Brown
James Stegall
Karen Keiller 🥌 📚
David August ❌👑
Tristan
Led By Gilded Fools
Lstn2urmama 🇨🇦
Steve's Place
Video Processing Wiki
publictorsten
Joe Stewart
zl2tod
Gordon W
JamesLundblad
FreediverX
tyil