Tim AllisonJul 20, 2023

Now another esteemed project/repo I can wreck... 🤣🤣🤣

But seriously, thank you @sebnagel and the entire #ApacheNutch team for this invitation. I'm honored to join.

Thank you!

Tim AllisonJun 8, 2023

It would be great to add an #OpenSearch 2.x indexer into #ApacheNutch.

Anyone able to help make progress on: https://github.com/opensearch-project/opensearch-java/issues/181 ?

Cc @OpenSearchProject

Ref: https://issues.apache.org/jira/browse/NUTCH-2994

[FEATURE] Reimplement BulkProcessor · Issue #181 · opensearch-project/opensearch-java

Are there any plans on adding an equivalent of the BulkProcessor API, which was available in the High Level Rest Client, so that Index and Delete requests can be batched?

GitHub
Show threadTim AllisonMay 21, 2023

@digitalpebble sadly homegrown one off: https://github.com/tballison/file-observatory/tree/main/commoncrawl-fetcher

If I were to do it again, I’d use #ApacheNutch or #StormCrawler

file-observatory/commoncrawl-fetcher at main · tballison/file-observatory

Single server/laptop grade file-observatory. Contribute to tballison/file-observatory development by creating an account on GitHub.

GitHub
Show threadTim AllisonApr 19, 2023
@willoremus I ❤️ that Google uses #CommonCrawl and thereby the fruits of #ApacheTika and #ApacheNutch.
Tim AllisonMar 24, 2023
@elan also see #Heritrix and of course #StormCrawler as alternatives to #ApacheNutch
Show threadTim AllisonMar 23, 2023
@elan #CommonCrawl runs #ApacheNutch
Tim AllisonFeb 24, 2023

#OpenSearch #ApacheNutch index writer on its way?

Please take a look and offer feedback!

I started with #OpenSearch 1.x.

https://github.com/apache/nutch/pull/761

NUTCH-2920 -- first working attempt at an OpenSearchIndexWriter by tballison · Pull Request #761 · apache/nutch

…iter to OpenSearch Thanks for your contribution to Apache Nutch! Your help is appreciated! Before opening the pull request, please verify that there is an open issue on the Nutch issue tracker wh...

GitHub