Mastodawn

Rob Ricci May 11

There's a lot of stuff going around about datacenters, so I decided to do a quick tour yesterday of some of the datacenters in the Salt Lake Valley. Some are indeed quite large, but there are a bunch of smaller ones too - and they are not always where you think!

All of these are publicly known, and you can find them (and ones in your own area) at https://www.datacentermap.com/ .

Let's start with a datacenter that I go by all the time! It's across the street from my grocery store in downtown #SLC. It's listed as a colocation facility; datacenters are famously secretive about who their tenants are, but we can guess that it probably hosts servers belonging to nearby businesses, especially ones that want their storage, etc. nearby, but don't want to have to maintain a secure, cooled room. Given the number of banks that have headquarters nearby, I'd bet at least some of them are customers.

This is a fairly little guy, with apparently 16k square feet of floorspace and 1.6MW of power.

Rob Ricci May 11

Next, an even smaller datacenter, that just about anyone in #SLC has seen! This is XMission, a local Internet Service Provider that's been running since 1993, so one of The Ancients in Internet time. It's on a very busy part of 4th South, and if you've been by at night, you've seen the big LED display on the front of the building that they put various animations on.

One of the things that I *think* is probably in this building is SLIX: https://slix.net/traffic/ - this is an Internet Exchange Point (IXP), where various carriers meet up to exchange traffic without it having to travel long distances. These are often run as a sort of community infrastructure - it's in the best interests of all networks involved to connect to each other so that they can do their jobs more efficiently.

SLIX is fairly small (according to their own data they carry ~200Gbps, with some spikes up to 1Tbps). There are about 40 networks that meet there: https://slix.net/participants/ . Funny story, when I first got Google Fiber at my house, I was getting routed through California to get to the University of Utah campus just a few miles away. I pinged a guy I know who pinged a guy he knows who ... learned that some of the participants in SLIX didn't have their routes set up right. A config change later, and not only me, but basically everyone on any commercial ISP in the Salt Lake Valley had much more direct routes to campus!

Rob Ricci May 11

This one is larger, physically (22.5 sq ft), than the first datacenter we looked it, but claims less power: 490kw. That's not a ton of power - my Chevy Bolt can draw 150kw from its batteries at max acceleration, and there are much bigger and sportier EVs that can draw something almost up in the range of this datacenter! (though only for brief periods of course! this datacenter probably draws a substantial fraction that much 24/7) Why is there so much less power for this datacenter?

Well, one of the key factors of datacenters is how power-dense they are: how much power they are designed to deliver to each rack, and how much heat they are capable of moving out.

Compute - especially GPU compute for AI - is incredibly power-dense and incredibly hot. So we can guess this datacenter is probably not for compute. If I had to guess, this is probably mainly intended as a "carrier hotel" - it's probably focused on having telecoms companies as tenants. I base this both on the lower power density, and where it is: it's near the Utah State Fairpark, which is in turn relatively close to train tracks heading both east-west and north-south. A lot of long-distance fiber in the US follows both the rail and Interstate road networks, because it's relatively straightforward to run fiber alongside transportation links. Salt Lake City lies on the west side of some of the few passes through the Rockies, so it has a ton of fiber, following I-80, the Union Pacific, etc. This is a good place for carrier hotels.

How is a carrier hotel different from an IXP? At an IXP, the carrier is just pulling in some fiber, maybe one or two routers or something. But they have a lot more equipment that they need than that - they have servers of various kinds too, plus the bigger backbone routers that fan out in many directions, etc. Mobile carriers have a fair amount of wired topology to deal with. That's the kind of stuff they put in carrier hotels, and this is a good spot for them.

I picked this shot because, in the background, you can see the Gadsby Power Plant, one of the main sources for power in #SLC. That's a natural gas plant that generates about 300MW. Put a pin in that number, we'll come back to it later.

Rob Ricci May 11

Now we're getting a bit bigger, and also more residential. This one sits on the edge of a residential neighborhood, on 200 E, in Milcreek. This is a 36k sq ft, 1.9 MW facility. What's in there? I don't know, as mentioned above, datacenters don't tend to tell you who their tenants are. There's probably some reasonable computing power in there, but it's probably not dense enough to be very GPU-heavy.

The sounds of the HVAC systems were quite noticeable at this one. Any time you are dealing with electricity, you are also dealing with heat. In a datacenter, the power drawn by the compute and network equipment gets turned into heat, and you need to get rid of it. Of course, you want to spend as little electricity getting rid of heat as you can. Datacenters call this "Power Usage Effectiveness", commonly called PUE. A PUE of 1.5 means that for every KW that goes to computers, .5 KW goes to other stuff - mostly cooling, but also heat losses, lighting, etc. A 1.5 PUE is pretty good, supposedly some of the biggest datacenters have PUE of around 1.1 .

This actually highlights one way in which having a fairly large-scale datacenter is efficient: putting all the computers in one place does enable you to use cooling systems that get rid of more heat for less power. Of course, how many computers you have, where your power is coming from, what mechanisms you use to cool them, etc. matters too! Again, we'll get back to that later.

By the way, my guess would be that only the building in the front is a datacenter - the building in the back has too many truck bays and not enough cooling. It's probably a small warehouse of some sort.

Rob Ricci May 11

This is the biggest datacenter I visited - campus, actually. This facility is in West Jordan, near the South Valley Regional Airport. It's big enough that I have to post several pictures to get you a real sense of the size (but it's not the biggest datacenter in Utah.)

What you're looking at here is three buildings that, together, have a power capacity that's reported (depending on the source) to be around 160 MW (put a pin in that number too.)

Two of these buildings are multi-tenant (the ones with the flat white roofs) like the others we've seen.

That third one in the back, with all of the cooling towers on top, has supposedly been built for a single hyperscaler, and is supposedly something like an 80-100MW building. Which hyperscaler? That information is not public. That's a whole lot of cooling on the roof (which is reported to be water-free), so my money would be on this being an AI data center.

In these pictures, you can see more electrical infrastructure. Bringing that much power into one place takes a lot of wires.

Ty Jones May 12

The reason I went on this little tour was to put in perspective the proposed Stratos datacenter project in Box Elder County, UT.

Stratos is supposedly designed to eventually reach a size of 9 GW. That is more than double the 4 GW that the entire state of Utah currently uses. The entire campus is supposed to be big enough that, for comparison, it would fill over 10% of the Salt Lake Valley, as shown in this image (which I didn't make).

That last datacenter campus? At ~160 MW, those three buildings put together are designed for a load about 1/55th the size of Stratos. That 300 MW natural gas power station we saw in the background? Stratos is supposed to generate its own power on-site, so it will need 30 of those things. (Or maybe more - remember PUE?)

In terms of carbon output, this thing is designed to be an absolute monster.

There's not much getting around that. They have handwaved about including solar and/or wind, but without anything concrete, we should assume this is a whole lot of carbon.

How about water?

Well, that's harder to tell, given all the vagaries and "if"s in the public information so far.

Remember, a datacenter has to get rid of a lot of heat. A datacenter that is generating its own energy on-site has to get rid of *far* more heat.

In the desert West, the most *energy* efficient way of getting rid of heat in the hot summer months is evaporative cooling: you boil water. This has, historically, been a major way of cooling both natural gas plants and datacenters, as well as homes, etc.

The same reason why this works well in the west is the same reason why it's problematic: we have very dry air, so evaporative cooling is very effective, but having dry air is connected to the fact that we don't have much water to begin with.

There *are* ways to air-cool natural gas turbines, and there *are* ways to cool datacenters that are not evaporative cooling. They are more *water* efficient. But they are less *power* efficient, which means, in this context, burning even more natural gas.

The backers of Stratos claim that they are trying to get some very new, high-tech gas turbines that operate without water cooling, or at least with very little. That does assuage some water concerns. But their language is very hedge-y - they're trying, they hope to jump in line for the limited supply of them, etc.

They also claim they will use "closed loop" water systems for cooling the datacenter. There are several things this *could* mean, and we need to know more in order to actually understand it. Most cooling systems for datacenters and even large buildings have a closed loop of water (or another coolant) for moving heat around. That's because we cannot *make* cold, we can only *move* heat. In some datacenters, this cold loop comes into the room, where it's used to cool air, which is blown across the servers. In higher-power-density datacenters, the coolant loop comes all the way to the individual rack in order to cool the air right before it enters the servers. In the most high-tech datacenters (which Stratos would likely be), it comes all the way *inside* the server, directly exchanging heat with the hot bits like CPUs and GPUs.

Coolant in these kinds of systems circulates, it's closed, you can generally consider the coolant loop to consume very little to no water after it's been filled.

But: you still have to make the heat go away somehow. This is where Stratos *might* use evaporative cooling. Or they might opt for one of the more expensive, less energy efficient dry systems. Saying "we have a closed loop" only tells us *part* of the story!

Here's what we know: the Stratos people have secured 13,000 acre-feet of water rights. In numbers that mean more to most of us, that's about 4 billion gallons per year.

They *claim* that's far more water than they need, and they won't use most of it.

But: if they don't manage to get their air-cooled gas turbines (which, in addition to being less efficient, also cost more), or decide to go with some evaporative cooling for the datacenter (because it's cheaper and uses less power), they could very easily use that much water. We are very much in a "trust me" situation, and it's not clear that we *should* trust what developers say when they are trying to get permits. We need to get independent studies and binding contracts.

For those who aren't locals, you might not be aware, but: the Great Salt Lake is shrinking. People are trying (not hard enough, probably) to save it. Not just because hey, what would we call our city without it, but also because the lakebed is full of chemicals we'd rather not be breathing in, thanks.

Stratos would not literally pull water out of the lake (which it is quite close to). But: the water rights they have obtained are in the watershed of the lake. So: if they use the water rights they have obtained, they might well contribute to the drying up of the lake.

The point here is that: they are hoarding water rights that they claim they will not use - the more reasonable bet is to assume they will use them; we need a study by actual hydrologists to understand whether using the water would accelerate the lake's demise.

And, you will notice that I have not even touched on a ton of *other* issues, such as:

1) Is there actually demand for all of these computers?
2) Would it be a good idea to fill this demand even if it does exist?
3) Can we build enough computers to fill this thing in a reasonable time anyway?
4) How far will this project get before the AI bubble pops, and will it leave anyone other than the investors holding the bag?
5) If it does get fully built, what other resources (like more water rights) might they go after?
6) Is it a wise idea to provide huge tax breaks to companies that expect to be highly profitable?
7) This is being done though the Military Installation Development Authority - what's the actual military connection here?
8) Regardless of whether it's wet or dry, is dumping this much heat into one valley a good idea?
9) There's no way that burning that much natural gas doesn't raise gas and electricity prices.
10) Can we trust the developers' numbers for how many jobs this will create locally?

Just to name a few.

Rob Ricci May 11

Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.

Rob Ricci May 12

I was on the local news about this, lol my office is such a mess

https://ksltv.com/science-technology/closed-loop-cooling-systems-save-water-but-can-be-a-drain-on-electricity/907589/

Closed-loop cooling systems save water, but can be a drain on electricity - KSLTV.com

While closed-loop cooling systems, like the one being touted for a large data center in Box Elder County can save lots of water, they often use more electricity in return, which can impact the environment in other ways, according to Dr. Ricci, a professor in the University of Utah's school of computing.

KSLTV.com

Katrina Katrinka

@ricci
Interesting! So do you use distilled water for the closed loop?

Rob Ricci May 12

@katrinakatrinka

I don't know the exact level of purity they go for, but yeah, removing things that could leave mineral deposits or cause corrosion is important

It is often mixed with glycol to lower the freezing point (no idea what Stratos would do, they have given us nowhere near that level of detail)

Katrina Katrinka

@ricci
I use a CPAP and was thinking of the kind of water I need in that. Adding something to lower the freeze point is also interesting.

RichardInSandy May 12

@ricci I am reminded of a story that I think Kurt Vonnegut told about his brother. Someone commented that the brother’s desk was a mess. He gestured to his head and said ‘If you think this desk is a mess, you should see what it’s like in here’.

Rob Ricci May 12

@richardinsandy my collection of spherical objects is clearly visible, dunno what that says about what's in my head

RichardInSandy 6d ago

@ricci I still have facebook to stay in touch with family in the UK. Second item in my feed this morning was KSL5’s interview with you, Rob.

Rob Ricci 6d ago

@richardinsandy so sorry for inserting myself into their lives in this fashion

Ashton Wiersdorf May 12

@ricci …something about a clean desk being the sign of an empty mind…

Rob Ricci May 12

@wiersdorf This is a good question, so I looked up some numbers.

Two different sources get me something like 3 acre-feet of water per acre for alfalfa in that area: a 1994 report from Utah State: https://waterrights.utah.gov/docSys/v912/a912/a912044e.pdf (see SNWV in Figure 2), and a listing of a huge farm for sale in the Snowville area now: https://www.land.com/property/6034-acres-in-box-elder-county-utah/4545825/ - claims 3895 acres under irrigation using 11.7k acre-feet of water rights.

So that would mean that the Stratus project has secured enough water rights to farm about 4.3k acres of alfalfa (which is about 10% of the land they say they have access to).

So: this is not nearly enough water to farm all of that area with alfalfa (is all of it even suitable for this purpose? no idea), but enough for a big chunk of it.

Of course, Utah is already watering vastly more alfalfa than we can afford to with our limited water resources.

Marcos Dione May 12

@ricci wait, most if not all DCs use open loop cooling systems?

Rob Ricci 6d ago

@mdione Most (probably all) DCs will use *a* closed loop where they circulate coolant (probably water, maybe mixed with glycol) to get heat out of the room (or directly off the chips). From there, many use systems that consume water to get that heat out into the environment. It's relatively new that large datacenters are trying to use entirely waterless systems on that side

Rob Ricci 3d ago

Addendum to my datacenter thread: This afternoon, I took a drive down south to check out some more datacenters in northern Utah.

This first campus is fully built out: it's on 23 acres with 5 buildings. The location is right off of I-15; if you've driven between Salt Lake and Utah Valleys, you've driven right by it. All five buildings combined are somewhere around 45 MW.

The campus was originally built by C7, a local datacenter company, but is now owned by Databank which owns datacenters all over the country. Databank is a US company, but it's owned by a bunch of investors, many of them foreign pension funds and asset managers.

As with many of the other datacenters earlier in this thread, this is a multi-tenant facility - the second photograph is from Databank's website, and shows how the inside of one of the buildings is divided by cages - tenants will have their racks inside of one of these cages, so that tenants can't access each others' equipment. In addition to being good security practice, this is needed for some regulatory requirements like HIPAA (health data) and PCI-DSS (payment card data).

Rob Ricci 3d ago

Okay, now the big boy - this is, so far as I know, the largest datacenter campus currently in operation in Utah. This is Meta's Eagle Mountain datacenter, which occupies something like 490 acres in Cedar Valley. It's much more remote than anything else we've looked at - it's in Utah County, but to the west, across Utah Lake and the Lake Mountains from Provo and Orem.

It's hard to be sure, but this datacenter seems to be in the 200MW-300MW range, based on public reports. Again, I'm posting several pictures, because it's hard to get a real sense of the scale of things thing.

Rob Ricci 3d ago

The campus is a few miles out of town, but Eagle Mountain itself is growing fast - largely in the form of developers buying up parcels of farming/ranching land and filling them with cookie-cutter subdivisions.

The address of the facility is on Pony Express Parkway - you may remember from earlier in the thread that fiber often follows earlier transportation networks. There's an old Pony Express station in this valley. Along many of the roads in the area (and all over, have a look and you'll see them everywhere) are these poles, which indicate the presence of buried fiber.

Rob Ricci 3d ago

This Meta campus is currently powered by the regular electrical grid; as you can see, this is a pretty serious electrical substation. Meta touts that it has helped to fund a few solar projects in Utah, enough to meet the campus's current demand. I don't know what the water usage is like.

However, this renewable energy story is changing. Construction is underway to expand the campus, and the city has granted the facility permission to build an on-site natural gas plant. I don't think we actually know how big the expansion or the natural gas plant will be, but they are probably at least a few hundred MW to make it worth doing. You can see here that they seem to be getting the construction site ready.

Rob Ricci 3d ago

Oh but wait, there's more. If you were looking closely at some of the previous pictures, you might have noticed another construction site in the background. This is QTS's SLC1 campus. Look, it's on Hyperscale Way. Cute.

There are three buildings under construction, with, apparently, two more planned. It's on a 193-acre site.

I can't find reliable information on the power consumption this is designed for, but my best guess is that these three buildings are at least 200MW; there are some sources that claim 650MW for all five, but I'm not very confident in that number.

The reason for the 200MW number is this: QTS has obtained permission for a gas turbine and/or fuel cell generation onsite, which seems to be up to 200MW. It's unclear at this point whether they are actually going to build it: originally, this datacenter was claimed to use grid power, then it was going to use this on-site generator, now it seems to be back to grid power. We'll have to wait to see what actually powers it.

Note that QTS claims to use a cooling system that doesn't consume water after the initial fill. (By the way, this was the same company that recently caused residents' water pressure to drop in Georgia during an initial fill there.)

Oh, and whose stuff is in here? This is probably starting to sound repetitive but: we don't know. QTS does build-to-suit, meaning that some other company is the one actually using this, probably as a single tenant. Is it more Meta? Is it a different hyperscaler? We don't know.

Let's bust out the comparison to Stratos again. Let's generously call the existing Meta facility 490 acres and 300 MW. Stratos would own 84 times as much land (though to be fair, they probably won't build on all of it). And at its full proposed size, it would draw something like 30x as much power.

If we add up both Meta and QTS, Stratos would be 60x bigger of a landowner, and draw somewhere between 10x and 22x as much power, depending on which estimates you use.

Stratos, if it gets built up to the maximum planned size, will be big. Real big.

Rob Ricci 1d ago

For the second addendum to my datacenter thread, we're going someplace that has no datacenters.

Yet.

That's right, get in, we're going on a field trip to Hansel Valley, unincorporated Box Elder County, Utah. Location of the proposed Stratos hyperscale datacenter.

Rob Ricci 1d ago

Hansel Valley is about a two hour drive north of Salt Lake City. It's in the Basin and Range geological province. This area, which stretches more or less from Salt Lake City, across Nevada, and into eastern California, is characterized by a bunch of north-south mountain ranges with valleys (basins) between them. The earth's crust has been stretched here, causing the cracks that drop to form basins and rise to form ranges. Remember this province.

The valley is about 30 miles long and 15 miles wide - the main road connection into the valley is about 30 miles from the town of Tremonton, and 40 miles from the only thing in the area big enough to be called a "city," Brigham City.

There are a few ranches, but that seems to be about it for human development. The only paved road makes it a few miles into the valley, then turns gravel. It's the kind of place where people spray-paint "Keep Out" signs on tires and where every sign has been used for target practice. To get there, you take I-84 up near the Idaho state line, then turn south.

I can count the number of people I saw in the valley on one finger (and he was nice and neighborly, so I won't use *that* finger).

Rob Ricci 1d ago

And here we are. Based on the map provided on the Box Elder County Stratos website, we're sitting right on the northern boundary of the main (southeast) area, looking south towards the Great Salt Lake off in the far distance. The borders of the project area are irregular, but I think that pretty much everything we can see here from the road over to the mountains, and most of the way to the horizon, is in the project area.

Here's the area where, in Phase I, they'd like to put 3 GW (the whole state uses 4-5 GW today) of electricity production and consumption.

Later, they'd like to triple that to 9 GW. Would it all be here? Or would some of it be in the other large project area 10 miles to the west? I'm not sure. Are they? If they are they haven't told us.

Rob Ricci 1d ago

Here's one of the major reasons they're interested in this site: this is the Ruby natural gas pipeline, which runs through Hansel Valley on its way from Wyoming to Oregon.

(Other reasons probably include the amount of land available, the presence of a state agency that can override local land use regulations, and the fact that they've negotiated tax breaks of 90% on energy use, 85% on the real estate, and 100% on the contents of the datacenter.)

What we're looking at here is a block valve station for the Ruby pipeline; this is one of the places the line is brought above ground for maintenance, monitoring, and emergency shutdown. (This station is actually in the next valley to the west, Curlew valley.) This pipeline has had a rocky financial history, and appears to have significant spare capacity. (Is it *enough* capacity? Is there enough production to fill it? I don't know.)

Rob Ricci 1d ago

Here's another major on-site resource for the Stratos project: water.

Part of the south end of Hansel Valley becomes salt marsh, then leads to salt flats, then out to the Great Salt Lake.

The developers of Stratos applied to change the use of about 600 million gallons per year from agricultural use by the Bar H Ranch for electrical generation and cooling - in the latter case, mostly for filling a system that they say will be completely closed loop, with occasional flushes of that system. The Bar H Ranch is also supplying a lot of the land, so it seems that Stratos has or plans to buy them out.

What we're looking at is downstream from the water right in question - the diversion point itself and area of current use are on private land, but hopefully this captures a sense of what the area is like.

Rob Ricci 1d ago

And here's the concern: the Great Salt Lake is shrinking. It's an endorheic lake, meaning it has no outlet to the ocean. There's a large area of the interior western US that drains into the lake rather than the ocean. It covers a bunch, but not all, of the basin and range province I asked you to remember earlier.

This is why it's salty: water, with dissolved solids, comes in from an area of 21k square miles, and only goes out via evaporation, leaving the solids behind.

We're already taking more out of the system than we put back in, so we end up with less lake. This is getting to crisis levels, to the point where there are various predictions that the lake will be gone in X years, with X varying by who's making the prediction and what the snowfall has been like recently. Point is, it's bad, and recently the state has been making some efforts to try to get more water into the lake.

Oh, and there's more stuff in the lakebed than just salt. There's a bunch of heavy metals and other stuff that's poisonous. Some of this is natural, some of it's from industry, and some is from agriculture. But if it's deposited as dust in a dry lakebed, it'll become airborne when the wind comes through, and we'll all be breathing it in.

And let's be clear: datacenters are not the main problem. Agriculture, especially alfalfa for livestock feed, is a much bigger issue, as is watering lawns. But you can understand why people are touchy about anything that's going to consume more water that would go to the lake.

You can just barely see the water in the distance in this shot - the lake is very shallow, so small drops expose a lot of lakebed.

Rob Ricci 1d ago

A lot of people complained about the change application, which to be clear covers a small fraction of the water rights the project says they have access to. One of the protests came from the Center for Biological Diversity, which is well-placed to understand the ecological and legal issues surrounding water use.

The developers withdrew it immediately. Will they file it again? Will they commit to *actually* using the waterless gas production and closed-loop cooling systems they've handwaved about, and give up on the water right? Their withdrawal letter claims they intend to continue the project and will re-submit with more information, so we'll just have to wait and watch.

Oh, and it's worth mentioning: water rights in Utah follow a 'use it or lose it' rule. So while they *say* they don't intend to use all of that water, it's worth considering: IANAWRL (I am not a water rights lawyer) but my understanding is that if they apply for this change, then don't use it that way, they will lose the water rights. So we should assume they will use everything they are applying for.

I'm illustrating this post with two things: the first is an observation well near Locomotive Springs, about 10 miles from the Hansel Valley site. The Utah Division of Water Rights uses this well to monitor aquifer levels. The others are shots from Bar M Springs, part of the Locomotive Springs complex, a couple miles from where the water would be diverted. It may share the same aquifer as the spring that's proposed for use by Stratos. This is a wetland used by migratory birds; it's already in poor shape, probably due to groundwater pumping in Hansel Valley.

Rob Ricci 1d ago

Don't forget that the Stratos project site has two large chunks of land (plus a third location right off the freeway, probably for support businesses). Here's the other one; I'd describe it as being in the next valley to the west, Curlew Valley, because I had to drive around a mountain to get there. But the Stratos flyer on the county webpage doesn't distinguish it as being in a different valley, and it's surprisingly hard to tell from maps. Kind of feels like the sort of place that locals know, but is an afterthought re: documentation on most maps.

Anyhow, here it is. This is taken from what I believe to be the southwestern corner, on Locomotive Springs Road; I believe we're actually seeing less of this one than we saw of the other. Still, as you can tell: it's massive.

There is also even less here, and it's more arid. There was next to nothing in Hansel Valley. There's even less here.

What are the plans for this spot? It's extremely vague.

Rob Ricci 1d ago

So, what to take away from this? Where a datacenter is located, the local resources, and its scale determine its impact. What it would displace, the infrastructure that needs to be built to access the site, make a big difference, as well as what it would extract from and put into the local environment.

Rob Ricci 1d ago

Just one more thing: I lied earlier. We can have a little datacenter, as a treat.

On the way out, I stopped by this little guy up above I-84 in Rattlesnake Pass (just about a mile from the head of the Hansel Valley).

It's probably not a datacenter per se, more likely a 3R station for long range fiber. Long distance fiber loses signal gradually over distance, and every once in a while you have to amplify it, reshape the beam, and/or adjust the timing. That's probably what's going on in this building. There's also a very large tower across the freeway that has mobile antennas and microwave dishes, it's also possible that there's some add/drop equipment or even mobile carrier equipment in here.

You can't see them all in this picture, but it has a number of cooling units (probably dry, I doubt they'd drag a water line up here just for this) and an on-site diesel generator. Seems to be normally powered by an electrical line, with the diesel for backup.

Bob Young 1d ago

@ricci
That was a very informative thread, thank you so much. About the last picture, the one with the tower. I can speak to that a little bit. That’s a very standard rural cell site, supporting three systems. I get paid to inspect those.

Each system is at it’s own level on the top-hat. Two systems could be owned by the same carrier – for example, their 4G system and their 5G system. Or, it could be two different carriers’ systems. T-Mobile and Verizon are often co-located on the same tower.

Quite often, none of the carriers own the tower. A tower management company like Crown Castle will lease the land from the landowner – often BLM, State Forestry, or similar – put up a tower, and lease tower space to multiple tenants. Some tenants may be private companies, or more often government agencies like State Police. The top-most of the three systems has fewer antennas and may be one of these. Or, the top-most system may be a minimally operated 3G system, often legally mandated to serve older phones as long as there are still paying subscribers with older phones in the coverage area served.

The microwave dishes lower on the tower are the ins and outs that connect the equipment at the site to the rest of the network. Sometimes rural sites are daisy-chained by microwave links in places where there’s no fiber or copper connectivity to the rest of the phone network.

You mentioned that the tower is across the Interstate from the Level 3 Communications fiber facility. That makes sense; the carriers would love to have fiber close by. I tell my students that “the dirty little secret of the wireless world is that everything is connected to the wired world on the back side.” Fiber, more often than copper, is the back side connectivity these days.

Rob Ricci 1d ago

@fifonetworks awesome, thanks for adding this context!

Karl Statz 1d ago

gaytabase 15h ago

@ricci now i am no expert but i wouldn't have thought salt water would be the best thing for cooling computers

Rob Ricci 15h ago

@dysfun they did say something about having to purify it before they can use it

@ricci the tsmc plant in Arizona is incredibly huge. Maybe not even as big as what you’re talking here but still just a lot of developed land.

It looks like the desert floor is being terraformed in a game.

Rob Ricci 3d ago

@elebertus whoa, looks huge. Amazing the size of buildings needed to manufacture things at 4 nanometers

@ricci so much infrastructure. It is crazy

Steve Purcell 2d ago

@ricci Ed Zitron calls out what I think might be this specific Utah datacenter as "not going to get built" in this video: https://m.youtube.com/watch?v=TCeXwFWmv1U

Ed Zitron: You are being lied to about the AI data center boom

YouTube

Rob Ricci 2d ago

@sanityinc I also think it's highly unlikely to get built, *certainly* not at the 9GW size

Commodore 2d ago

@ricci how many futures can you go back to with a 9GW facility?

Rob Ricci 2d ago

@commodore hey man, if they they made that movie today, and went back the same number of years, guess when they'd go to?

Commodore 2d ago

@ricci don't do this to me man, my psyche cannot take it!!

Dr. Eric J. Fielding, PhD May 11

@ricci Thanks for the detailed description. I saw something about the residents trying to fight this but didn’t know the crazy scale.

Rob Ricci May 11

@EricFielding Also, a thing I didn't mention is that this is not the only datacenter of this size being proposed in the state. There's another one just as big being discussed for central Utah.

M.S. Bellows, Jr.May 11

@ricci This is REALLY thoughtful and informative; thank you. (And it's worth saving/sharing even outside Mastodon, so: hey, @mastoreaderio ! Unroll!)

Masto Reader May 11

@msbellows here's the unrolled thread: https://mastoreader.io?url=https%3A%2F%2Fc.im%2F%40msbellows%2F116557139885627239

Next time, kindly set the visibility to 'Mentioned people only' and mention only me (@mastoreaderio). This ensures we avoid spamming others' timelines and threads unless you intend for others to see the unrolled thread link as well.

Thank you!

Masto Reader

Jaime Herazo May 11

@mastoreaderio
Well, this is handy
@msbellows

M.S. Bellows, Jr.May 11

@jherazob @mastoreaderio Ain't it? I just wish it wouldn't scold me every time I trigger it publicly. I WANT both the request and the resulting link to be public so other people can learn about/benefit from them!

Gerbrand van Dieyen May 11

@ricci naive question, wouldn't building large amounts of solar panel be more energy efficien

Rob Ricci May 11

@gerbrandvd I don't know the exact math on this, unfortunately. What I do know though is that you'd need both solar *and* storage, in a setting like this where they're generating all of their own power on-site, they'd need to generate far more power than they need during the brightest hours of the day, then use it overnight and/or when it's overcast.

Then there's also the fact that one generates far less power from solar during the winter when the days are shorter and the angle of the sun in the sky is less favorable (and that you'd have to clear the panels of snow).

Solar might be a reasonably good match for the cooling part of the load; you can sometimes get away with using outside air when it's cool enough (winter and sometimes at night) but would be a lot harder to make work for the actual computing load, since that's going to run 24/7/365 (esp. if this is used for AI training)

Phairupegiont May 11

@ricci Thank you for the overview.
What I don't understand is, why build data centers in areas with warmer climates, when colder ones would be... well, easier to cool?

Aren't economic and ecologic incentives aligned here?
Data centers for compute in particular (as opposed as, for response time) don't need to be in any particular geographical area anyway, do they?

Rob Ricci May 11

@phairupegiont You are correct! It's generally easier to cool things down when it's colder outside! Here in northern Utah some datacenters - with far less power density than this one - are able to just use outside air to cool for a good chunk of the year. With the kinds of heat loads generated by warehouses of GPUs; well, I suspect their cooling needs are indeed lower in the winter, but they probably need active cooling all the time anyway.

There are some datacenters in Finland that even use cold seawater as part of their cooling systems!

So, why build in places that get hot part of the year? Well, if you are willing and able to use water for evaporative cooling, that's pretty effective in very dry environments - and can be cheap depending on the cost of water. Sometimes, the availability of power is a big thing too - in this case, there is an existing natural gas pipeline running through the valley that they intend to tap. For some kinds of datacenters, it's important to be near your users - though that's less important for AI training, which is what this one would likely be used for.

@ricci @phairupegiont how about heating up the rocks instead of the air? It's about +20C inside the mountain, and nobody would complain if you make it warmer.

Rob Ricci 2d ago

@bonkers @phairupegiont ground-source heat pumps are definitely a thing, though I've never heard of a datacenter using them

@ricci @phairupegiont digging would be expensive, and of course the water is cheaper.

@ricci @phairupegiont it kinda doesn't belong to anyone...

Rob Ricci 2d ago

@bonkers @phairupegiont unsure if this is meant as a joke but taking it seriously, water rights are a *major* thing in the western US and in fact part of the reason they have (maybe temporarily) withdrawn their application to use the water in the area is that the rights holder may have not been using the water, which, under Utah law, means they've lost the right and can't give it to the datacenter. It'll be interesting to see this play out

@ricci @phairupegiont rather, a joke. My point was rather, those who live there, and also the nature, don't have much influence on those decisions.

bryan newbold May 11

@ricci great thread, thanks!

Moe Lassus May 12

@ricci great thread, Rob. I worked for one of the big cloud providers and got to spent quite a bit of time with the datacenter team before I retired. I was blown away by the innovations being implemented to reduce power and cooling requirements. All of that is moot now. We were talking about the potential of 25kW racks. Now they’ve completely blown past that with 100kW racks. It’s insane.

@ricci great thread, it really puts things in perspective!