OpenSearch 3.1 has arrived, bringing a wave of innovation to boost indexing performance, refine search results, deepen observability insights, power advanced agentic AI, and enhance the security of your deployments. Dive in to learn about all the new release features announced today.

To learn more, see the blog here: https://opensearch.org/blog/get-started-with-opensearch-3-1/

#opensearch #observability #agenticAI

Get started with OpenSearch 3.1 - OpenSearch

OpenSearch 3.1 brings further innovation into the 3.x line with a host of updates to help you boost indexing performance, improve search results, dig deeper into your observability data, build more powerful agentic AI solutions, and maintain security for your OpenSearch deployments.

OpenSearch
Standing room only for Andrew Ross's Keynote this morning at Open Source Summit. Telling the community about #OpenSearch 3.0 and Lucene 10, and how it's Paving the Way for AI and Vector Innovation. #OSSummit #opensource
On stage now Dotan @horovits talking through the journey #OpenSearch went through From Fork To @linuxfoundation #OSSummit #opensource
Come by the #OpenSearch booth at Open Source Summit and let Adam Tackett give you a demo on the #OTel capabilities in OpenSearch. #OSSummit
Hello Open Source Summit - Denver! Looking forward to speaking with you this week. #OSSummit #OpenSearch #opensource

Malcolm v25.06.0 includes a some new and oft-requested features, bug fixes, and component version bumps.

Compare v25.05.0 to v25.06.0

NOTE: As this Malcolm release enables the OpenSearch Security Plugin as described below, even inter-container access to OpenSearch must now be authenticated when using Malcolm's embedded OpenSearch instance. To accomplish this, an internal-use-only account and password is used for connecting to OpenSearch by Malcolm's other components as needed. This credential (saved in .opensearch.primary.curlrc in the Malcolm installation directory) needs to be generated before Malcolm starts up the first time after upgrading. To do so, please run ./scripts/auth_setup and select (Re)generate internal passwords for local primary OpenSearch instance. This credential is only used internally for OpenSearch and cannot be used to remotely access Malcolm.

  • ✨ Features and enhancements
    • This release adds role-based access control (RBAC) to Malcolm (cisagov/Malcolm#460).
      • Malcolm's RBAC feature is based on Keycloak realm roles and is implemented in to layers:
      • Whenever possible, Malcolm's backend Keycloak realm roles are mapped to the roles/groups/permissions features provided by the components that make up Malcolm (see release notes for details)
      • For other Malcolm components that don't implement their own permission management systems, Malcolm handles the enforcement roles based on request URIs in its NGINX proxy layer.
      • This is an optional feature. RBAC is only available when the authentication method is keycloak or keycloak_remote. With other authentication methods such as HTTP basic or LDAP, or when RBAC is disabled, all Malcolm users effectively have administrator privileges.
      • Because the OpenSearch Security Plugin requires TLS even internally, Malcolm's internal connections to the embedded OpenSearch instance, when used, are now all performed over HTTPS. However, this is all handled internally and should not behave or appear different to the user than it did in previous versions.
      • See the role-based access control documentation for more information on this feature.
    • Malcolm's embedded KeyCloak instance now automatically creates and configures the default client by ID, if specified in ./config/keycloak.env.
    • Allow user to specify subnet filters for NetBox autopopulation (cisagov/Malcolm#634)
      • This feature is especially useful for excluding dynamic address ranges such as those used by DHCP, which should generally not trigger autopopulation in NetBox. Since these addresses can change frequently and aren't tied to specific devices, including them could result in inaccurate or noisy inventory data. By fine-tuning which private subnets are included or excluded, users can ensure that only meaningful, typically static assignments are autopopulated.
    • Expose init arguments for Arkime's db.pl and also use them for Malcolm's creation of its own index templates (cisagov/Malcolm#692)
    • Extend Zeek's intel.log with additional fields using corelight/ExtendIntel (part 1) (cisagov/Malcolm#502)
      • This integrates the corelight/ExtendIntel plugin into Malcolm internally but does not significantly change how Malcolm presents intel.log to the user. Further work to do so will be continued in cisagov/Malcolm#695.
    • Some internal tweaks to the PCAP processing pipeline that are going to be leveraged by the Malcolm-Helm project (idaholab/Malcolm#630)
    • Handle a fix in the ICSNPP OPCUA-Binary plugin that adds a new sec_token_id field (cisagov/icsnpp-opcua-binary#101)
    • Moved the configuration for Zeek's use of the zeek-kafka plugin to its own file (kafka.zeek) to make it easier to override in Docker using a volume bind mount or in K8s using a configMap.
    • Changed some internal objects used for NetBox enrichment caching from Ruby's Concurrent::Hash to Concurrent::Map for better performance
    • Minor improvements to the icons, shortcuts, and convenience bash functions in the ISO-installed Malcolm desktop environment
    • NGINX now generates a robots.txt file to avoid web crawlers
  • ✅ Component version updates
  • 🐛 Bug fixes
    • NetBox autodiscovery no longer populating host name from DNS, DHCP, NTLM (regression, cisagov/Malcolm#699)
    • documentation served at /readme is trying to pull fonts from use.fontawesome.com (cisagov/Malcolm#694)
    • support fractional gigabytes correctly when generating Arkime's config.ini setting maxFileSizeG from PCAP_ROTATE_MEGABYTES
    • Improved logstash filters that calculate unique hashes used as document IDs for Zeek and Suricata logs to better prevent duplicate logs from being written to the document store
  • 🧹 Code and project maintenance
    • Tweaked some code comments and documentation to bring the cisagov and idaholab repos into harmony.
    • Documentation improvements
    • Removed some unused files and outdated comments

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

#Malcolm #HedgehogLinux #rbac #Zeek #Arkime #NetBox #OpenSearch #Elasticsearch #Suricata #PCAP #NetworkTrafficAnalysis #networksecuritymonitoring #OT #ICS #icssecurity #CyberSecurity #Cyber #Infosec #INL #DHS #CISA #CISAgov

Day 1 of #OpenSourceSummit kicks off in Denver.

Don’t miss From Fork to Foundation
🕕 6:00 PM MDT | Bluebird 3C
Hear how OpenSearch became a community-led project under the Linux Foundation.

Schedule: https://events.linuxfoundation.org/open-source-summit-north-america/program/schedule/
#OpenSearch #OSSNA #OpenSource #LinuxFoundation

Schedule | LF Events

This schedule is automatically displayed in Mountain Daylight Time (UTC -6). To view the schedule in your preferred timezone, please select from the drop-down menu to the right. IMPORTANT NOTE: Timing…

LF Events
PDX to DEN ✈️
See you at Open Source Summit
#OpenSearch #OSSummit #opensource

Cambi infrastrutturali in Uber: dai limiti raggiunti con #elasticsearch a contributori di #opensearch

https://www.uber.com/en-IT/blog/evolution-of-ubers-search-platform/

The Evolution of Uber’s Search Platform

Search is a foundational pillar of Uber’s user experience, directly influencing key aspects of our core business. Whether you’re looking for a meal on Uber Eats or requesting a ride, search is what connects you with the right options—quickly, accurately, and at scale. Uber Eats users are faced with a staggering choice: over one million restaurants globally, and typically more than a thousand dishes or restaurants per user session. Helping Uber Eats users navigate this vast space—based on preferences like cuisine, price, delivery time, dietary needs, or past behavior—is a deeply complex problem.

Uber Blog

For any #MTCP, #DOS and #Retrocomputing nerds out there who are also running httpserv and want pretty graphs, poke me for a recipe for a hideosly bloated #logstash configuration to ingest the UDP logs.

I feed it to #Graylog which stores the data in #Opensearch - a pipeline that combined (and this is accurate) needs, conservatively, 4096 times as much RAM as the floppy museum itself (8MB).

And while looking at this when making this screenshot: I wonder why someone would hit http//floppy.museum with a Referer-header indicating they come from a salesforce-dot-com address? http-colon-slashslash-136.146.46.127 (about halfway down the list).

#msdos #bloatware #theremustbeabetterway