Hilko Bengen
Kenneth FinneganOne of my nice friends at Hurricane Electric gave me a dead 100G-LR4 optic to tear apart for your entertainment, so for the sake of your entertainment, lets dig into it! 🧵
100G-LR4 is a QSFP28 optic that runs over a duplex pair of single mode fiber, so it takes 4 lanes of 25G from the switch, modulates 4 different colors of light, and combines them together over a single strand to the other end up to 10km away.
First comes off the ejection bail. I guess we're committed now since that was rather destructive.
Two tiny screws get us into the case. The grey putty stuff is thermal gap filler, which couples each part that puts off heat to the case for cooling.
Think of it like thermal paste, but it's designed to be thick and span a gap instead of just be a thin film.
Lifting the electrical assembly out of the other half of the metal case, we can start pointing out the major points of interest.
TOSA/ROSA - transmit/ receive optical sub assemblies are where the laser magic happen.
The retimers help 4x25G get to the switch via the QSFP connector
The TOSA and ROSA both need to handle four wavelengths of light moving 25Gbps each, so they're actually connected to the PCB with two layers of ribbon cables. One of the ribbons is just for the 4x25Gbps data channels, and the other is for power, biasing, etc, not high speed stuff to run the OSA parts.
Looking on the other side of the whole assembly, the largest IC in the middle (the silver one) is the microcontroller that talks to the switch over an I2C bus to report telemetry and ID information, and pretty much everything else is voltage regulation for taking the 3.3V from the switch and regulating it down to lower voltages to drive the lasers.
And that microcontroller that runs the whole show is... a Cortex M3! An STM32F103C6 with 32kB of flash, 10kB of SRAM, and can run at up to 72MHz.
Meaning that this optic has more compute power than many early home computers.
You'll note that the ST microcontroller is silver, as are a few other ICs on this optic. This is actually a cost saving measure called "flip chip" packaging, where they don't waste the expense of the black plastic packaging and solder the silicon die straight down onto the board like it was a QFN. Absolutely wild.
While it's fun to think about malware on the ARM core in these, the data path is WAY faster than the Cortex M3 can handle. It's entirely an out-of-band manager of the electronics with no idea of what is going on in the Ethernet link itself.
That being said, some newer optics really DO have interactions in the data path. The most notable is the speed-changing optics which allow modern Ethernet switches (which only support 10G/25G/50G on their front panel ports) to still link up with a 1G Ethernet peer. The optic does the 10:1 speed change and implements clause 37 autoneg entirely inside the optic.
Granted, the firmware on optics actually *IS* field writable. I've coordinated with an optics vendor in the past to help a customer apply a firmware patch to 10,000 optics installed in Arista switches in the field by issuing the right magic sequence of write commands over the I2C bus from EOS.
Remember how I said the rest of the parts were mostly voltage regulation?
Voltage regulators usually need some kind of inductance... so just saying... prime cute coil of wire opportunity.
So this is where things get REAL ugly... we break out the Dremel and cut the lid off the TOSA (Transmit Optical Sub Assembly) to expose the yummy optical bits inside it.
Remember that while there's four different waves for the four 25Gbps streams, 100G-LR4 runs over a single duplex pair of fibers, so all four optical signals need to make it out a single strand of fiber, so... PRISMS.
The colors are all way out in the infrared, so don't put too much stock in the colors I use in the picture, but hopefully you get the point about how the prisms are used as optical combiners.
The keen eye will also notice that some of the Dremel debris from me cutting open the TOSA is... stuck to this cube and fuzzy... like it's magnetic...
And it it. That is a tiny magnet in the middle of the optical path, but why?
Physics talk. Remember that the transmitters here are lasers, which mean that the four lasers on the right are each their own resonant cavity, which leaks a fraction of the light to be useful. But if there was some kind of partial reflection out in the fiber somewhere, this could cause problems and destabilize these lasers, so we would like some way to let light OUT of the TOSA without letting any reflected light back into the lasers to cause problems, so we make an optical one-way valve...
MORE PHYSICS TALK: Light is a perpendicular E-M field, so if you used a vertical polarizer, and then passed the light through a magnetic field to rotate the polarization 45 degrees to line up with a //// diagonal polarizing filter on the other side of the magnet...
Think about the reverse path. //// polarizing filter, 45 degree rotation, horizontal light hitting a vertical polarizing filtering. BLOCKED.
Putting the TOSA under my microscope, we can start to see some more of the finer details.
This is rotated from the last photos.
I'm not convinced whether the two missing lenses in the middle broke off before I got it, broke off while I was hacking my way into the package, or were just never there. A mystery for the ages.
For one thing, we see a tiny square and two bond wires landing on the platform holding the lasers and the first stage of the optics.
This is a silicon diode.
Fun diode fact: the junction voltage of a diode changes with temperature in a very predictable and consistent way, so it's very common for temperature sensors to measure the junction voltage of a diode or base of a transistor.
And a temperature sensor would imply some kind of temperature control, and this actually makes a lot of sense, because the four lasers are all different colors, but only SLIGHTLY different colors.
Each of the four lasers are only 4.5nm apart in wavelength, so the temperature really matters to keep them in their channel, so there's a peltier cooler underneath the platform holding the lasers and first optical stage so that the optic can electrically heat or cool the lasers to keep them at the right temperature!
The ROSA is somewhat less interesting. The right half seems to be a giant prism and I wasn't able to get it apart to any useful degree.
The glassy fractures on it are from me trying to pry it out.
The four receive diodes on the left of the ROSA are facing up, which is perpendicular from the light coming in from the right, so the final stage of the ROSA optics were glued as a bridge with four little lenses and a right angle PRISM!
So lastly, here are some more photos of the PCB that runs from the QSFP edge connector to the ribbon cables attached to the ROSA and TOSA. 🧵 /fin
Thanks for reading. For those new here, I have a Liberapay which I'm using to fund the #MicroMirror project where I'm trying to fix Linux/free software distribution to users. https://liberapay.com/phirephly/
Pixel Refresh@kwf Images this gory should come with a warning
@PixelRefresh Fair. But really, I should just come with a warning in general.
Sander@kwf Although I did not understand all of it, this was fun to read. Very interesting as well. Thanks a lot!
Łukasz C. Jokiel@kwf Great show, thanks!
@lukaszjokiel 🎉 Here to entertain!
Stacey Campbell@kwf I used these over 100m at a previous job, they were flawless. Nice to know how the sausage is made!
Peter William Wright@kwf thanks for the breakdown & explanation, it was a fascinating to read.
Michael Potts (HMHackMaster)@kwf Incredible thread; this is legit @foone level stuff and I am loving it!
I had no idea about the peltier cooler stuff, but it makes total sense! Do you know if that exists on more plebian 1G/10G optics? I have always been fascinated/concerned by how hot SFP modules get but it all seemed within spec. Now I am wondering if the heat is the optic trying to cool itself which might mean a bit of attention to airflow to keep them cooler might actually be beneficial.
@HMHackMaster @foone 1G/10G are a single color, so exact wavelength control is much less important for them.
CtzKane@kwf awesome thread! Thanks for sharing!
Kronsby@kwf probably my favorite thread I have seen on here. Thanks for posting!
NeoFox@kwf Hmm, interesting how different the ROSA is to the TOSA. I would have thought that they would have identical optical pathways.
Am I right in guessing that the receivers are sensitive to very specific wavelengths, so it doesn't matter that the right beam hits the right detector, so it just scatters the entire beam over every detector?
@NeoFox The receivers are broadband, so you need very selective optical demux to break apart the signal. Either with a diffraction medium like a prism or often a diffraction grating based splitter.
Phil Dennis-Jordan@NeoFox @kwf I assume that’s why that last prism is so big? Presumably a cascade of prisms or something like thin film optics internally to pry those wavelengths apart? I mean if you think about how little say the red and yellow in white light diverge after going through a prism, and there’s about 150nm or 20% wavelength difference there - single digit nm deltas in 1000nm-ish light (<1%) must be pretty tricky to separate.
Jerome
Mark Theascetic@kwf: Cooler! Wow! Now I need to learn how to use emoji on Mastodon. 😮😮
> a peltier cooler underneath the platform holding the lasers
Thanks, that was fun!
I really didn‘t expect the Peltier, now I am wondering in what other tiny parts they may be hidden and whether they can be a feature of the surface structure nowadays.
Attie Grande@kwf: Note the angles.
gunstick@kwf these may be crystals doubling or halving the frequency of the light. So you only need 2 different types of lasers. And one couple of beams is simply double of the other ones.
@gunstick The four lasers are on 4.5nm spacing since they're using the standard LWDM grid, so they're probably just alignment optics.
sen@kwf I think they're optional guides to ensure the lasers hit the prisms correctly as the placement of the optics doesn’t look to be very precise. If you’re cutting costs you could have a human on the assembly line placing them as well as they can and the additional lenses are only added if they don’t pass QA.
Zorin =^o.o^=@kwf This stuff is MIND BLOWING. I recall your thread on birdsite a few years back when you tore down an earlier generation 100G optic that used TEN different wavelengths!
Seems the tech has moved forward; now we just need four!
@zorinlynx Well... NOW we only need one. 100G-LR is taking over the industry since it doesn't need the optical muxes anymore and just uses an electrical gearbox to combine the 20 PCS lanes from 4x25G to 1x100Gbps.
@kwf Of course this means there's probably an optic combining ten wavelengths at 100G each to put 1T on a single strand because that's how it rolls. :)
@zorinlynx Well... try 8x400Gbps, but you've got the right idea.
Trec 
🦊Obvious in hindsight, but I never thought of magnetic fields being able to change the polarization of light, neat trick
Fabian Laasch ⚡️🔋@kwf Witchcraft! :D
Neratas Frosthorn Stormwing@kwf this, oddly, given the understanding that photons are transmissed EM fields, that have the capacity to alter EM fields where they arrive, makes me question. Is perhaps, our understanding of the Strong and Weak force flawed? Mayhap there is a particle like a photon that leaves quarks, changing their flavor, and likewise changing the flavor of quarks their radiation ends up interfering with? Sorry, I barely understand quantum physics, but I love wild speculation.
The Uberduck@MisterStormwing @kwf photons have the unlimited range they have because they're massless. The bosons that mediate the strong and weak interactions have mass, which limits the range of those interactions to more or less the width of a nucleus.
@MisterStormwing @kwf also, there's not a whole lot of stuff that the current understanding of quantum chromodynamics and electroweak theory doesn't explain; it's gravity that we still haven't quite nailed down. If you're interested in this stuff, the PBS Spacetime series on YouTube does an incredible job explaining things.
Z̈oé@kwf wow! TIL
datenwolf@kwf The Faraday effect (here used to implement an optical isolator) is quite versatile. Consider the following problem: You want to measure the current through a high voltage transmission line (think >>200kV). Obviously measuring the voltage across a shunt resistor is riddled with so many problems, you'd rather avoid it. You could of course use a current transformer, but to maintain the isolation gap its opening would have been so large, that it also couples a lot of stray fields. 1/2
@kwf So what can you do? Well, optical fibers happen to be excellent electrical isolators. Electric currents and magnetic fields go hand in hand. So instead of winding an electrical conductor around the transmission line, you wind some length of Faraday active optical fiber around, send polarized light through it and measure the amount of polarization rotation on the other end: That's an optical current transformer. https://en.wikipedia.org/wiki/Fiber-optic_current_sensor
@datenwolf That is awesome and I had never thought about it.
^,,^ 
@kwf Specifically the interaction of a magnetic field and light in some matter through the Faraday effect, magnetic fields themself won't have any effect on light (theoretically unless they're very strong, practically almost unachievable).
VE7FIM@kwf Is that grey rectangle on the left a magnet?
Adam Thompson@kwf nicely done! Had to zoom WAY in to tell those were prisms!