Adam ♿Mar 3

Hey @jpm, what kind of fibre and optics do I want? going to do 2-4 ~10m runs.

I am currently getting in way over my head with a 25 gigabit Mikrotik switch but this started as point to point runs.

Think future-proof, not necessarily cheap but of course some of this shit is going to come #StraightOuttaShenzen

Looks like I might have painted myself into a bit of a corner with some "cheap" SFP28 gear, too, but I'm not too attached

Show threadI love this, so IMar 3
@voltagex remind me tomorrow, but in a nutshell go for 10km 1310nm or 1550nm single-mode if it’s not too much more expensive than 850nm multi-mode. You’ll be able to find everything you need on FS.com
Show threadI love this, so IMar 4
@voltagex right, brain engaged, i think, so let's talk high-speed ethernet and the SFP family of optics.
Show threadI love this, so IMar 4
@voltagex (after picking up Little Miss 7 from school early because she fell on her face)
Show threadI love this, so IMar 4
@voltagex Ok, SFPs. Little rectangles, about THAT big, one end goes into network-related thingo, other end goes into cable
Show threadI love this, so IMar 4
@voltagex and that's about it. There's no magic to them, they just translate one signal to another. On the host-side (switch or NIC), it's a high-speed serial connection, and on the cable side it could be anything, limited only by the imagination of a SFP vendor and market demand.
Show threadI love this, so IMar 4
@voltagex In general, you'll only really need to know about SFP, SFP+, and SFP28, and only really need to know about them in the context of Ethernet. QSFP+ and QSFP28 are just 4 SFP+ or SFP28 in a trenchcoat.
Show threadI love this, so IMar 4
@voltagex so, what's up with the cable end? That's where things get deep into "it depends" territory. Luckily, it boils down to 3 different cable families.
Show threadI love this, so IMar 4
@voltagex namely, single-mode fibre, multi-mode fibre, and twin-axial copper (think, coaxial cable, but with 2 cores instead of 1). Even better, the OG 1Gbit/sec Ethernet SFP doesn't even really have the twin-ax option, it only really started appearing from 10GbE.
Show threadI love this, so IMar 4
@voltagex The twin-ax option is the simplest, but the most limited. It literally just electrically connects the high-speed serial links on the host equipment together. Given how fragile these high-speed serial signals are, the cable length is extremely limited to about 10M at most, and anything over about 5M is going to have active signal conditioning and amplification onboard. It works great for anything inside the same rack or the immediately-adjacent rack, but don't go too much further than that.
Show threadI love this, so IMar 4

@voltagex the second option is "multi-mode" fibre optic cables and optics. Traditionally, they were cheaper than "single-mode" but this is no longer true, because cheap VCSEL lasers are now capable of single-mode emission. They operate in the 850nm near-IR light band, and cables are generally constructed from 50um diameter plastic or glass cores. Because of the loose tolerance of the optics and fibre cores, their range is usually limited to 300M, and there's multiple grades of multi-mode fibre available with corresponding limitations on distance vs speed similar to copper ethernet cables. If you're buying new now, just go with OM4. OM1 cables use 62.5um diameter cores, and are only usually found in up to 1Gbit/sec applications. OM2 and later are all 50um diameter cores, but the optical transmission properties vary between generations which will limit transmission length.

You definitely don't need the fancy hybrid single-mode optics that use multi-mode cables unless you're trying to run 10GbE up the side of a building with ancient OM2 multi-mode cable plant.

Show threadI love this, so IMar 4

@voltagex the third option is "single-mode" fibre optic cables and optics. This is where Shit Gets Real, and you do not look at energised optics with remaining good eye. Luckily, the actual cables are already so good and cheap that you just buy whatever from a reputable vendor and it'll work with any new single-mode optical devices you ever want to attach in future.

The optical devices on each end, however, are another matter entirely.

The basic consideration is the range class - typically either 10km, 20km, 40km, 80km, or 120km. Direct correlation to how much power the laser uses, and how expensive the modules is. Faster speeds may not get the 120km range class. Yes, you can have a single piece of glass 120km long to carry your signals.

And then comes the "oh wow holy shit my mind is blown" consideration of WAVELENGTH DIVISION MULTIPLEXING

Show threadI love this, so IMar 4
@voltagex unlike electrical signals where only a single signal can be present on a conductor at the one time, you can send MULTIPLE DIFFERENT SIGNALS, IN DIFFERENT DIRECTIONS, AT THE SAME TIME, ON THE ONE FIBRE CORE
Show threadI love this, so IMar 4
@voltagex you don't NEED to, of course, but you CAN, just by replacing the optical components on both ends.
Show threadI love this, so IMar 4
@voltagex the "basic" single-mode SFPs run at either 1310nm or 1550nm far-IR wavelength, and use a single dedicated fibre core for each transmission direction. Not really any different than a multi-mode fibre connection, except the distance is now at least 10-20km
Show threadI love this, so I
@voltagex the "bi-directional" single-mode SFPs come in pairs, where one of the pair transmits on 1310nm, the other in the pair transmits on 1550nm, and both transmit on a single fibre core at the same time. The receive path uses a wavelength-selective prism to direct the received wavelength to the receiver inside the SFP module while passing the transmitted wavelength through with little attenuation. This is the basic theory of how GPON networks work over a single fibre core as well.
Mar 4 at 4:27amWeb
Show threadI love this, so IMar 4

@voltagex and finally the "wavelength division multiplexing" single-mode SFPs specify their wavelength, which is then optically combined into a single fibre core and transmitted to the other end, where the multiplexer uses wavelength-selective optics to split the signals back out into separate wavelengths on multiple fibres again.

Oh yeah, and there's 2 kinds of WDM, "coarse" and "fine", but they both work the same way, the only difference is how closely the wavelengths are packed together.

Almost certainly you don't need this, but you can just attach WDM MUXes to the single-mode fibres you already have whenever you want!

Show threadI love this, so IMar 4
@voltagex and for extra lols, you can combine CWDM optics and muxes and BiDi operation, so you end up with 8 x 25Gbit/sec full-duplex links running across a single fibre core.
Show threadI love this, so IMar 4
@voltagex and finally, 10GbaseT SFP+ modules are an abomination and should never be used if you can help it, because they generate so much heat and eat so much power it's not funny.
Show threadAdam ♿Mar 4
@jpm yes, although I've had to literally burn myself on them a couple of times to learn this
Show threadI love this, so IMar 4
@voltagex As for where to buy this stuff, just go to https://fs.com/ I run their stuff in production environments, and they're just as good as the vendor-blessed optics at 1/10th the price. They also have a bunch of exotic optics that vendors don't even offer, but work perfectly well because SFPs are just an abstraction layer between layer 2 and layer 1
FS.com Deutschland - Rechenzentrum, Technologie-Lösungen für Unternehmen und Internetdienstanbieter

Bereitstellung skalierbarer Netzwerktechnologie-Lösungen für Rechenzentren, Unternehmen und Internetdienstanbieter. Entdecken Sie Hochgeschwindigkeits-Netzwerklösungen zum Aufbau einer zuverlässigen und skalierbaren Verbindungsinfrastruktur. Weltweite Lieferung, professioneller Support.

FS.com
Show threads0 Is Not A SysadminMar 4

@jpm @voltagex important to note though that they work by spoofing the manufacturer codes that switches check for because vendor optic lock-in is where they make cash money. So you can’t just throw any old FS optic in a Cisco and expect it to work — you have to get one with the Cisco coding.

Oh yeah and the Linux drivers for intel 10G cards default to enforcing this “vendor supported SFP” shit unless you explicitly set a kernel parameter because open source capture by corporations lol

Show threads0 Is Not A SysadminMar 4

@jpm @voltagex oh hey Joel want to see my favourite things that render your last sentence hilariously inaccurate when it comes to weird edge cases in broadcast/AV industry?

  • DirectOut USB.MADI operates a high channel count USB sound card converter on FPGA in the SFP
  • Riedel MuoN HDMI/SDI to/from lossless broadcast Video-over-IP converter
    • Show threads0 Is Not A SysadminMar 4
    @jpm @voltagex we’re over here on layer 2+4i of the OSI model
    Show threadI love this, so IMar 4
    @s0 @voltagex I’ll just stay squarely inside the OSI model with these perfectly spherical cows
    Show threadI love this, so IMar 4
    @s0 @voltagex but that’s the neat* thing about SFPs, they can mangle any data stream to/from a high speed serial data stream
    Show threadJustified Ancient of MoominMar 4
    @jpm @s0 @voltagex or low speed - I’ve recently been testing out little SFP pluggables that are actually single-port terminal servers for OOB management.
    Show threadI love this, so IMar 4
    @ThermiteBeGiants @s0 @voltagex SGMII that links up at 1Gbit/sec always runs at just over 600MHz…
    Show threadJustified Ancient of MoominMar 4
    @jpm @s0 @voltagex oh while we’re talking about weird SFPs - there’s also ones that convert SGMII 1Gbe into LCAS’d STM-16 SDH out there, should you wish to run your traffic into an SDH ADM with some TDM traffic smushed into it
    Show threadJustified Ancient of MoominMar 4
    @jpm @s0 @voltagex what I’m saying is, maybe I should start a Weird SFPs account, but that might be stepping on Nanoraptor’s toes a bit
    Show threadI love this, so IMar 4
    @ThermiteBeGiants @s0 @voltagex nah, real things are definitely fine, eg the cursed footprints account
    Show threadvia unreachableMar 4

    @ThermiteBeGiants @jpm @s0 @voltagex do iiiit

    (if you do, https://blog.benjojo.co.uk/post/smart-sfp-linux-inside is another good one;

    as are those VDSL2-modem-on-an-SFP mostrosities)

    Ghost in the ethernet optic

    Show threadvia unreachableMar 4
    @ThermiteBeGiants @jpm @s0 @voltagex ooh, I'd heard about those for 10G (10GBase-LW etc.), but never for 1G!
    Show threadJustified Ancient of MoominMar 4
    @oddline @jpm @s0 @voltagex https://www.smartsfp.com/documents/TSoP_STM16_productbrief.pdf they’re weird beasties
    Show threadbelee flys wayy too muchMar 4
    @ThermiteBeGiants @oddline @jpm @s0 @voltagex If you want to play with a MuoN A10 or A1, I'm sure i can find one kicking about for a day or 2
    Show threads0 Is Not A SysadminMar 4
    @belee @ThermiteBeGiants @oddline @jpm @voltagex omg what about this fun one? https://www.rad.com/product/smart-sfps/miclk-grandmaster-on-an-sfp/
    MiCLK Grandmaster on an SFP

    MiCLK 1588 Grandmaster by RAD offers precise PTP and GPS timing in a compact SFP. Ideal for 5G and LTE-A deployments to support IEEE.

    RAD
    Show threads0 Is Not A SysadminMar 4
    @ThermiteBeGiants @oddline @jpm @voltagex I love to read a Wikipedia article and basically everything in it is completely new to me
    Show threadJustified Ancient of MoominMar 4
    @s0 @oddline @jpm @voltagex if you want to learn about “dead” telecoms technology I’m your guy apparently
    Show threadI love this, so IMar 4
    @ThermiteBeGiants @s0 @oddline @voltagex it’s not dead, there’s still customers paying 6 figures annually for a service on that line!
    Show threadDaniel CarosoneMar 4
    @jpm @ThermiteBeGiants @s0 @oddline @voltagex fibers installed along the tops of HV transmission lines, originally just to detect breaks if towers fall, later found themselves being used to carry data. Lots of SONET/SDH.
    Show threadJustified Ancient of MoominMar 4
    @uep @jpm @s0 @oddline @voltagex GSM-R train radios, as the GSM heritage in the name suggests, use TDM too - E1 into SDH ADMs and then terminating direct into the BSCs as SDH (apologies for the initialism soup there)
    Show threadJustified Ancient of MoominMar 4
    @jpm @s0 @oddline @voltagex nah, the telcos have gotten rid of almost all of their TDM service now, it’s not even a legacy cash cow for them. You either get a leased MPLS-TP service (which IMHO is a big step down from TDM in many ways) or you roll your own physical network - none of the telcos even lease dark fibre to you unless you’ve got a deeeep chequebook
    Show threadI love this, so IMar 4
    @ThermiteBeGiants @s0 @oddline @voltagex eh? Dark fibre is still definitely a thing and isn’t even that expensive, only a couple thousand a month from Vocus or PipeNetworks for CBD building connectivity.
    Show threadJustified Ancient of MoominMar 4
    @jpm @s0 @oddline @voltagex for proper regional backhaul though?
    Show threadI love this, so IMar 4
    @ThermiteBeGiants @s0 @oddline @voltagex depends on carrier, Vocus claims to have some* regional** backhaul. Optus flatly refuses to sell dark fibre though
    Show threadJustified Ancient of MoominMar 4
    @jpm @s0 @oddline @voltagex dollars to donuts that the Vocus dark fibre is sparkling MPLS
    Show threadI love this, so IMar 4
    @ThermiteBeGiants @s0 @oddline @voltagex their metro dark fibre is a single piece of glass fused together from point A to B
    Show threadvia unreachableMar 5
    @ThermiteBeGiants @jpm @s0 @voltagex can confirm, I would pay embarrassing sums of money for TDM service, but they're all getting out of that game as fast as they can
    Show threads0 Is Not A SysadminMar 5
    @oddline @ThermiteBeGiants @jpm @voltagex it’s so irritating how hard it is to get dark fibre hookup
    Show threadJustified Ancient of MoominMar 5
    @oddline @jpm @s0 @voltagex closest you can get is CESoP over an MPLS-TP tunnel but the tunnel bit is the crucial bit there - you need equipment at both ends.
    Also it is very funny to have to have a 2.048Mbps E1 frame carried over 10Mbps+ of MPLS-TP provisioned pseudowire
    Show threadMalcolm HerbertMar 4
    @jpm @voltagex there's another variant you haven't mentioned - the SFP serial port ... https://www.youtube.com/watch?v=XgmQ-qBrqxY
    The SMALLEST Serial Console Server

    YouTube
    Show threadAdam ♿Mar 4
    @mherbert @jpm I'm an idiot so I have the SFP VDSL modem somewhere
    Show threadMalcolm HerbertMar 4
    @voltagex @jpm hahah ... using SFP for anything other than fibre feels ... Wrong
    Show thread"Musty Bits" McGeeMar 4
    @jpm @voltagex aha! So the promise of SPF+ for 10Gbe in a home setup is bs. Good to know.
    Show threadnoodleMar 4

    @arichtman @jpm @voltagex SFP+ DACs here and very happy.

    Edit: but this is probably the turning point where I decide to run 25gbs when I move my NAS et al out of the livingroom.

    Show threadAdam ♿Mar 4

    @arichtman No?

    I will disagree with @jpm here - only because I've tried some different gear. The Aquantia AQC113 NICs are fine, haven't tried the new Realtek 10 gig NICs but they are also reasonably low power.

    SFP+ DACs are fine, 10 gig copper SFP+ is fine* if you have enough space around them for cooling.

    https://www.aliexpress.com/item/1005009139011834.html

    https://www.omadanetworks.com/au/business-networking/omada-accessory-media-converter/mc420l/ - this one in particular is great and doesn't get hot. @jpm happy to send you one to check out.

    KeepLiNK 10-Port Netzwerk-Switch | 8 x 2,5 GB Ethernet + 2 x 10 GB SFP+ | Unmanaged | Plug & Play | Hochgeschwindigkeitskonnektivität - AliExpress 7

    Smarter Shopping, Better Living! Aliexpress.com

    aliexpress.
    Show threadJulien GoodwinMar 4
    @jpm @voltagex there's an optical networking 101 talk that gets run at NANOG every few years, it's worth watching (or at least reading the slides)
    Show thread"Musty Bits" McGeeMar 4
    @jpm @voltagex what the fuck lmao
    Show threadI love this, so IMar 4
    @arichtman @voltagex they’re just different coloured lights
    Show thread"Musty Bits" McGeeMar 4
    @jpm @voltagex THATS THE POINT - is this not witchcraft to you? Yes they vibrate at juuuust the right period so they don't interfere.
    Show threadI love this, so IMar 4
    @arichtman @voltagex not really. It’s the same as 2.4GHz and 5GHz WiFi sharing the same air
    Show threadDaniel CarosoneMar 4

    @jpm @arichtman @voltagex and the same as CWDM mux

    e.g. random example:

    https://www.fs.com/au/products/214981.html?now_cid=177

    16 Channels CWDM Mux or Demux, 1310-1610nm, with Monitor and Expansion Port, 4.5dB Typical IL, LC/UPC, Single Fiber, FMU Plug-in Module - FS.com Australia

    16 Channels CWDM Mux or Demux, 1310-1610nm, with Monitor and Expansion Port, 4.5dB Typical IL, LC/UPC, Single Fiber, FMU Plug-in Module, Product Specification:Channel - 16 Channels , Wavelength - 1310-1610nm, Special Port - Monitor, Expansion Port, Technology - TFF (Thin Film Filter), Insertion Loss - ≤5.0dB, 4.5dB Typical, Channel Spacing - 20nm

    FS.com
    Show threadI love this, so IMar 4
    @uep @arichtman @voltagex yep I did touch on those later in the thread. The wildest ones are BiDi CWDM with 8 bi-directional channels on a single fibre core
    Show threadDaniel CarosoneMar 4

    @jpm @arichtman @voltagex the perils of seeing the thread via a reply and not the main line, and not a tree view.

    Also, "dense" rather that "fine", but yes.

    The cool thing with these is that because they're a purely optical device, each wl can be a completely different L2. we used to do a bunch of Ethernet and FC over different colours between failover sites.

    Show threadDaniel CarosoneMar 4

    @jpm @arichtman @voltagex

    Oh, and the other discussion about STM across power transmission reminded me of another important use case for CWDM.

    If you have multiple colours going down the same fibre, you can immediately tell the difference between a link break caused by a failure in the SFP optics and a fibre cut.

    Because the CWDM box is an entirely passive unit, mostly a prism, it's rather unlikely to be the point of failure

    Huge benefit that IMHO justifies using the mux for that alone, when one of the ends is hard to get to.