I was compelled to explain in a large training session that Excel and MathCAD are not, Not, NOT acceptable tools for performing QL-1 nuclear safety calculations.

I absolutely hate that I have to keep explaining this to degreed engineers doing safety-critical work.

I am going to die on this hill whether I want to or not.

This just makes me want to cry.

Excel is suitable only for tasks less important than Girl Scout cookie sales tracking. Do not bet anything more valuable than a box of Thin Mints on Excel.

@arclight
Excel is not a spreadsheet. Excel is a full-featured virtual machine running a smalltalk-inspired REPL whose display layer happens to resemble a spreadsheet.

Something like a third of the world’s money goes through Excel every single day, and the reason you don’t think Excel is a Real Programming Language is because if we admitted that, we’d have to admit that most of the most important software in the world was written by underpaid women in pink collar jobs, and we can’t have that.

@mhoye
Do you have a ref for excel-Smalltalk link? I'd be interested ... thanks!
@mhoye I think we're on the same page here. Excel is the world's most popular functional programming system. The problem is that it's sold and viewed as an easy-to-use tool for "non-programmers". The problem is that it is in fact a programming system, spreadsheets are software, and should be treated with the same rigorous review process as software written in traditional languages. Hiding the programming aspect of the system has a secondary effect of making them virtually impossible to verify.
@mhoye I can't really speak to the pink collar aspect; I'll have to forward that to my wife who is the Excel expert in the family ;)

@arclight my guy, you just referred to excel as a child’s toy only suitable for tasks less important than girl guide cookie sales and then referred me to your wife as the family excel expert.

Just tell her how little you think of her, don’t involve me in this.

@mhoye I think you're really misunderstanding what I'm saying and that we're probably in furious agreement.

My wife works in emergency management and is regularly called on for complex Excel tasks. She is _literally_ the Excel expert in the household.

It's not that I consider Excel a toy, I consider it a tool used far outside of its intended scope of use by people who are not treated as programmers, not given adequate training or tools for their work. That's a problem.

@arclight "Excel is suitable only for tasks less important than Girl Scout cookie sales tracking."

Even if you're right about nuclear safety calculations - you might be! - a statement like this is on a par with arguing that for safety reasons, the database should be blue.

@arclight @mhoye I shouldn't do this...I should resist...but umm...why do you refer to her as the "Excel expert in the household", instead of the expert in her job place?

I think that your intent is possibly thoughtful...but you may be using habitual language that is making things worse.

@kristinHenry @mhoye I don't know her workplace as well as I know our household (i.e. the two of us and a pair of cats). I know she does more complex work with Excel than I do; I'm not really in a position to claim she's the expert where she works. She's more skilled with Excel than I am. That's it; that's the whole message.
@arclight @mhoye The technical point you are making - that the lack of verification systems for Excel makes it unsuitable for applications where correctness is important - is not unreasonable and it's possible that it's true. The way you've talked about this though uses language that resonates with misogyny. If you don't want that, it's worth debugging what you have internalized that you can identify and unlearn. If you think it would be helpful, I can point out some of the cues I see here.
@cdemwell @mhoye I understand that; I've chosen my words carefully and am open to clarification and questions. However, I'm not interested in arguing over words I haven't said, positions I don't hold, or hair-splitting over terminology or tone. Sorry/thanks.
@arclight @mhoye No need to apologize to me. If you chose your words carefully then I just made wrong assumptions. I too am not interested in arguing, only in offering help, which isn't needed.

@arclight @cdemwell @mhoye I mean, you still haven't walked back "Excel is suitable only for tasks less important than Girl Scout cookie sales tracking."

That's what's holding up understanding here and preventing your message from being cleanly received.

@trombonejoe
"Excel is suitable only for tasks less important than Girl Scout cookie sales tracking." is not a statement that needs walking back because it it true. The misunderstanding is taking that statement to imply the people who are not given better tools are somehow at fault. i don't see that implication and the author has explicitly rejected it.
@arclight @cdemwell @mhoye
@arclight @cdemwell @mhoye - There is no escape for you in this thread. I recommend you change yer name and move to Belize. God bless you!
@MrShoggoth "As God as my witness, I thought turkeys could fly."

@mhoye I am biased, insomuch as I was a minor when I was first using Excel at version 1.0.

My dad used to have me write spreadsheets for his business. Unpaid of course (or risk being beaten & not having a roof over my head).

So while I sympathize with the un(der)rpaid part: please don't emasculate me, I contended with enough of that BS.

Excel has *limits*.

1,048,576 rows by 16,384 columns

Maximum file size for uploading to a document library: 50 (MB) default. 2 (GB) maximum
@arclight

@mhoye Contrast that with say, a "real" database.

I'll be kind & use SQLite: "The largest possible setting for SQLITE_MAX_PAGE_COUNT is 4294967294. When used with the maximum page size of 65536, this gives a maximum SQLite database size of about 281 terabytes."

So, no Excel is not suitable for "serious" work.

A lot of things are done in Excel.

Yet if you have more rigorous CS issues, please provide code with COMPLETE source.

That disqualifies basically EVERYTHING from M$, period.
@arclight

@byterhymer @mhoye @arclight i wish MY father beat me for NOT using Excel...

@arclight Excel isn't even ACID compliant.

You're not the only one calling out how it being misused leads to all kinds of problems either, also see:

https://www.datacenterdynamics.com/en/opinions/improving-data-integrity-wake-excel-controversy/

CSVs can be useful for lots of things very easy to twiddle with on a command line with sed, awk, tr, cut, grep, etc.

That doesn't mean that they are wise for many other use cases where more stringent methodologies should be considered.

@mhoye

@arclight @mhoye This.

There was, awhile ago now, a report out of Wall Street to the effect that any non-trivial spreadsheet is buggy in some important way, and while they had had every motivation and a lot of money to throw at the problem they just could not fix it.

Code is only as good as your tests, and it's nigh-impossible to test your Excel code.

@graydon That is the context of this thread. If you take your work seriously, you will use tools that support good, basic software development practices (e.g. testability, auditability, configuration mgmt). The initial post in the thread noted that the problem domain was engineering, specifically nuclear safety analysis. The argument applies to any work considered by its authors or their organizations to be important.

@mhoye @arclight

Ok, yes, but also no. It isn't just "little old lady in accounting" and that's a dodge. There are privileged white male engineers running giant spreadsheet programs determining the safety of the bridge you drive on with no way to replicate results if they leave the company.

@Okanogen @mhoye And that is precisely the case I'm faced with. There are other problems with how business, management, developers, etc. view Excel users. I'm principally concerned with engineers using Excel for technical work instead of a tool that can be easily verified for correctness. That's the original context of this thread.

@arclight @mhoye

"ZOMG! SOMEONE CHANGED THE VALUE IN '[concreteparameters.xlsx]#Constants!$CB$34' FROM "0.64" TO "0.32"! WHEN DID THAT HAPPEN? BEFORE OR AFTER WE ORDERED THE BRIDGE ABUTMENT MATERIALS?"
"UH, I don't know, it looks like that value is autgenerated from another sheet that was on Phil Miller's laptop (summer intern let go 4 months ago). Actually, it looks like all the constants are reduced by 50%....".
"FUUUUUUUCCCCKKKK! STOP CONSTRUCTION!!!!!!"
True story. Truthy, anyway.

@Okanogen @arclight @mhoye yeah this is the main issue with excel for sure (next to how easy it is for someone to open it up, mangle a random number of rows, and save it without realizing) and the only reason I'm not militant about it is that the state of other tools isn't too much better (outside of hosted and battle hardened systems whose job is to do and protect such stuff)
@mhoye @arclight I have long maintained that Excel is the gateway programming language. I had a friend who’d been an “analyst” for years claim that she couldn’t understand hard CS concepts. Until I explained them using the terms they were called in Excel.
@ContextSans This whole thread has been very interesting from where I started reading it. But @ContextSans it would be very interesting to see the Excel to CS comparison that you provided to your friend. You could probably write a book on that. (Note: I am not a developer or advanced Excel user.)
@tmichellemoore I think we talked about truthy variables, data structures (specifically how an array was basically another sheet and how you could refer to it in formulas), and then I think we got into database design and SQL -which are even more relevant but which I’m even less qualified to speak to.
@ContextSans I need to lookup truths variables, but that is understandable. Thank you!! 🙂
@ContextSans @mhoye @arclight Seriously. Excel is my default programming environment. It's easy to prototype some functions in cells with formulas, then move on to other programming languages—if needed. Even if I think I'm clever and can work something up in Python or C++, sometimes if I have to pass things on to other people in the organization, some "stupid" Excel spreadsheet is the right way to go. It's not about pride, it's about getting work done
@kittell @mhoye @arclight This makes me feel better about prototyping everything in Ruby Liquid 😆

@mhoye @arclight

A major problem with #Excel isn't anything to do with sex or age.

It's to do with domain applicability. "Everything is a nail."

The number of times that I've seen people do database work laboriously in Excel when _there's a database tool right there in Microsoft Office_ is depressing. I haven't yet personally seen someone do word processing or mail merge in Excel, but I would not be at all surprised by it.

Too often the problem is that of using only one tool everywhere.

@JdeBP

I had a boss in Human Resources who wrote entire Memos and letters in excel. They used the page break preview and never resized the cells, just worked within the page size. It was incredibly frustrating how hard they worked to NOT use the correct tools available to them.

@arclight @JdeBP @mhoye yes, but most of the time, those people are « business experts » in their domain. 10 years ago, I bought a stack of the « Manga Guide to Databases » and lend it to those business people to help them model their business data, and then to help us integrate/migrate those to relational database backends or ERP. It also eased the dialog with IT people as they better understood relational concepts and SQL.
@jrjsmrtn @JdeBP The context of this thread is well-trained, well-resourced (mechanical and nuclear) engineers intentionally using Excel when technically appropriate tools are available. If business domain experts are anything like engineers, they view themselves as subject-matter-experts-who-code, not as programmers. Asa result, all the developer guidance about testing, code review, version control, etc. never reaches them. "Why are you telling me this; I'm not a programmer." Yes, yes, you are.
@JdeBP @arclight ? I am a programmer, I was also at the time, a system integrator to be precise. I never said I was not :-)
@jrjsmrtn @JdeBP I was referring to the people you were giving the Manga Guide to Databases to.
@mhoye @arclight Yeah, and I've seen some of the results of spreadsheet code being written by people other than "real" software engineers (and not properly version controlled). Like, a quarter of a million pounds going missing. (White collar man in this case.)
@mhoye @arclight Excel is the tool you use to bodge a job when your bosses won't give you visual studio or the training you need to do it *properly*

@mhoye i thought it was a dodgy flight simulator…

https://youtu.be/-gYb5GUs0dM

Excel 97 Easter Egg - Flight Simulation

YouTube
@mhoye @arclight I've worked in around 9 industries over 3 decades and without Excel spreadsheets not one of those businesses could function.

@mhoye @arclight I agree with your assessment of the marginalization of some of the world's most important programmers, but *part* of that marginalization is that excel is not, in fact, fit for purpose, and they are functionally forced to use it.

Previously: https://www.nature.com/articles/d41586-021-02211-4

Previously: https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates

Previously: https://theconversation.com/the-reinhart-rogoff-error-or-how-not-to-excel-at-economics-13646

Previously: https://www.zdnet.com/article/excel-errors-microsofts-spreadsheet-may-be-hazardous-to-your-health/

Autocorrect errors in Excel still creating genomics headache

Despite geneticists being warned about spreadsheet problems, 30% of published papers contain mangled gene names in supplementary data.

@mhoye @arclight I think it's very easy to conflate "real programming language" and "good programming language".

@mhoye @arclight @otfrom the European Spreadsheet Risk Interest Group has a collection of gloriously horrifying stories starring Excel: https://eusprig.org/research-info/horror-stories/

Their conference is back this year; Chatham House rules and really fun/scary/interesting: https://easychair.org/cfp/EuSpRIG_2023

Horror Stories | European Spreadsheet Risk Interest Group

@mhoye

One of my previous jobs was helping a majority-male civil engineering team fix the problems with introduced by Excel.

What you *can* do with Excel is impressive, there's no question! The project I witnessed included 5,000 lines of BASIC. But you definitely shouldn't!

It's a really bad thing that most of the most important software in the world is written with no type checking and no unit tests.

@mhoye

But it gets worse than that because the typing is often almost completely unpredictable. We've all had issues with Excel aggresively changing things to dates, but a more insideous problem is serial numbers -> scientific notation. That destroys information. There's no way to get it back. You can render a whole Excel "database" useless just with that.

@mhoye

I'm a woman, and I definitely agree that female-coded jobs in STEM deserve more respect, but they also deserve better tooling than Excel. It shouldn't be used for anything important.

@webbureaucrat

@mhoye

I started my career pre-degree doing data mining and reporting in the ops department of UPS. Because we were not in IT or Industrial Engineering, we only had access to MS Access and Excel. I built an application that calculated the container equipment imbalance across the whole country and who needed to send empty containers where to prevent any areas running out. All in excel+access and some VBA macros. It's very powerful, but I would have killed for Java or Python.

@mhoye @arclight Not only that, (afaik, I only ever used Excel once because my school forced me to) it's also a purely functional language which means that those underpaid women in pink collar jobs manage to pull something off that even most devs can't do:
Using functional programming languages in production
@teleportaura @mhoye @arclight reactive functional programming languages no less.
@mhoye @arclight felienne hermans is a Dutch computer science professor who has been preaching this for many years. She even has a talk where she implements a sorting algorithm in excel live on stage. And I don’t mean in vba or something, but actually in the spreadsheet cells. Her phd research was about bugs in spreadsheets and how much they cost. The numbers were pretty frightening.
@mhoye @arclight @tess I remember many years ago when I thought I was a shit-hot programmer then having a woman I worked with in her 50s run rings around me with pivot tables - I still do not understand them - and thinking she wasn't technical enough. We definitely need to change the mindset - to me Excel is literally magic.

@mhoye one of the semiconductor test machines I work on uses test programs written entirely in excel. There’s a little gui where you load the parameters and then it spawns five to ten headless excel.exe processes in the background while it tests and bins all the die on a wafer.

I thought for sure the engineer that told me about it was joking but I’ve since had the pleasure of needing to get into the debugging environment and yup, it’s excel macros all the way down 🙃

@mhoye @arclight and every single year that excel has existed, CS techbros could have gotten over the command-line elitism and made GUI software that does everything excel can do, but with simple, configurable UI and performant databases & maths. But it doesn’t happen, at least in part because of this divide of “just learn a real programming language/GUIs are for noobs” that has always ignored the many benefits of GUIs (even simple ones) for both casual analyst programmers and end users.