I’m really happy with how my current project is going
This is an air quality sensor plugged into a microcontroller that’s transmitting readings over a mesh WiFi
I only have this one soldered at the moment, plus a prototype on a breadboard. That's my proof of concept that I could do this.
I learned lots doing this. Quite happy. Much to do still.
[edit] Code and documentation are at:
https://github.com/GuillaumeRossolini/griotte
Since this one works, I ordered a few more sensors and microcontrollers from the same manufacturers and I soldered them
I now have a bunch of these little devices around the house, and I am very happy with how they extend the range
As long as I place enough of them along the way, I can sprinkle them wherever I want and they’ll self organize, and one of them reports all their readings to an SQLite database on a RaspberryPi Zero
They are on their own WiFi, which doesn’t matter for security since I’ve written their program myself, but still feels great
Haven’t done statistics and graphs yet
Latest news is that I got my brain into embedded thinking mode
I changed the way I determine if a sensor reading is worthy of archiving
The way I set up my probes, each #ESP8266 send its readings as fast as it can (that’s every 3s) to the entire mesh and the root node, the #ESP32, forwards them across WiFi networks over HTTP to my #RaspberryPi Zero W
The rpi0 then decides what readings to keep and what to toss
The naïve strat
The way I was deciding what to toss was, until yesterday, an SQL query doing a GROUP BY on the node sending the readings and checking the MAX date for its readings
The database file is already several dozen MB in size and growing by almost that much daily (I’m stress testing my system),
Even with a proper SQL index, the SDCard has to read a lot of information on every incoming sensor reading (SQLite is a file based system, there is nothing in RAM), just to decide what to keep and what to toss. That translates into lots of unnecessary i/o of all kinds, faster SDCard wear, and general latency unpleasantness
The rpi0 CPU was 40% to 90% all the time, according to htop
I’m way too used to PostgreSQL…
The embedded-friendly strat
My new strat is simply creating a file on disk for each sensor, changing its modified timestamp and using that to determine when its readings were archived last
That’s just a handful of empty files on disk and I’m only changing their metadata, not their content. They’re using next to nothing in terms of space. Checking their metadata is also really fast, and possibly cached by the OS and/or my web server.
Now every HTTP request fits within 4ms if the previous reading for the same sensor was recent, or at most 80ms when the database gets a write
The rpi0 CPU is now back to low levels, between 7% and 12% most of the time
This morning my ISP had an outage and I restarted their router, which happens to power my rpi0 with the web server for this project
And I forgot to restart that web server, so I lost all data from today
But before that, it was performing beautifully. There were 844 records for each node since last night, like clockwork. I happen to have cleared my previous data last night, so all the nodes were running and already on a stable mesh at the time.
Each sensor has been broadcasting its readings every three seconds, at which point my server decides to ignore most readings until the 1 minute mark (per sensor) and I’m happy that not one minute was lost for half a day of recording
I am less happy that after the web server was restarted, none of the microcontrollers found it by themselves. It looks like the entire mesh collapsed because the root node couldn’t bridge both networks again.
I understand that the root node didn’t restart the bridge between both WiFi (I didn’t set it up that way), but I would have thought that the iot mesh should have continued independently? Perhaps because it is declared as the root node? I’ll have to test this more.
I reset the ESP32 root node and it picked up the home WiFi fine, but none of the probes connected to it again (this mesh should have self healed within minutes, at most)
I went around the house to reset each ESP8266 node and yes, that is when the mesh self organized again and I started seeing the web server collecting readings again, as their BME680 sensor finished its self calibration routine
#meshnetwork #AirQuality #esp32 #esp8266 #bme680 #raspberrypi
I’m very pleased with the general responsiveness of my little project
Pictured in the attached video is the bridge #esp32 node, the one that waits for data from the #mesh and forwards it to the storage device
The blinks represent data being forwarded from one WiFi to the other
Seeing 6 blinks within 3s means that all the probes have called home in time before their next sensor reading
Today I soldered a few more of my custom air quality thingies, flashed them with the exact same program as the other ones, gave them a USB adapter and plugged them in their own spot in the house
They automatically found their path in the mesh, then they started sharing and recording their readings, which automatically appeared in my dashboard, and all I had to do was rename them there
In the picture, the number in the parentheses is the number of readings for the day (more or less one per minute) and all bars and lines represent average readings for the day; I have one graph per room to get the timelines
Humidity and temperature are on the left axis, all the other are on the right axis
(Looks like the one in the bedroom crashed again)
The one in the attic is missing however long it took me to solder the new things, since that’s the one I unplugged to use as a template
Attached is a video of the #esp32 happily blinking its payloads across WiFi networks to the #RaspberryPi Zero, for storage in the #sqlite database
Thousands of readings per day (about 1300 each sensor so far, today), all sensors being within 15 readings of each other, I’d call that stable
I’m happy to notice that when a board fails, it eventually recovers on its own (it takes several hours, though)
This weekend I finally caved and used #ChatGPT [1] to improve a program I’m working on [2]
The project
To recap the above thread this is a part of, I’m trying to build a mesh network of microcontrollers (ESP8266), each with an air quality sensor and they broadcast their readings so that a different microcontroller (ESP32) can forward them to a Raspberry Pi Zero for storage, statistics, visualization etc.
I have about a dozen of nodes and the network has started to fall apart (it was fine when I had 6), so I’m trying to find workarounds but I am unfamiliar with C++. This is where ChatGPT comes in.
The experience wasn’t smooth. It felt a little like chatting with a junior dev, fresh out of school and full of ideas about how a happy path algorithm should be written or how an ideal library should interface, but less concerned with the code in front of them.
How it started
I started by sharing the public URL for the .ino file (it’s an Arduino IDE project) on my GitHub and it looks like the bot used it as a search rather than a link, and it scanned the top results but didn’t find my code. It told me so in so many words. I had to share the link to the raw code instead. No biggie but a bit underwhelming.
(As an aside, I find it funny that the bot’s default search engine, Bing, was unable to locate a file from a public repository on GitHub based on its full URL. Both of these services are operated by Microsoft.)
Then, having just seen the code and after I expressed my desire for a code review, the bot proceeded to explain to me generalities of programming and best practice. Meh.
First code review
After a third prompt from me, it finally read my code but still provided several comments about things that aren’t in my program. I don’t use a display anywhere, so why the advice about OLED? (I know why, that was in one of the results in its first search, the one where it told me it hadn’t found my code). Of course I am indeed using the data with an output, mine being the mesh rather than a screen, but… Meh.
Out of the six initial suggestions, only the one about the macro was useful; although with the amount of memory at my disposal, that particular optimization wouldn’t matter.
Of the other recommendations, none applied to my program and the bot kept suggesting their suboptimal variant as fixes later on, contradicting itself numerous times. Every time I corrected its mistake, it praised me for my insight and gave my a bullet list of pros and cons, which felt a little patronizing, and then it proceeded to make the same mistake a few prompts later. And again later on.
General impressions
The bot failed to refactor my code at any point, instead giving me basic unfinished examples modified to fit the current recommendation. One of the things I was worried about was my use of the JSON library and how strings are manipulated. After all, this is C++ and with a headless device that won’t tell me anything when it crashes. I tried a bunch of times to get help from the bot, failed and moved on.
Every time I challenged one of the bot’s suggestions, it came back to me with an apology and that after further review my code didn’t need that particular change or with a generic phrase to not even try giving me a firm answer.
The one suggestion that looked like it could improve things, about deep sleep, ended up being a non starter because that would power down the sensor and I would have to restart the calibration phase on wake-up (that’s about 5 minutes for this sensor). The bot didn’t know to check what consequences the change would have.
Mixing software and hardware
This part was interesting. I was challenging the bot’s assumptions about deep sleep and how this specific sensor would behave in that case, and I told the bot to go read the code of the library for the sensor. I had to prompt this twice because the first time it came back with guesses. It looks like it did actually search for it, scanned the top results and came back with apparently more insight into the inner workings of the library. It didn’t find the authoritative source that I was using, but that’s a tough ask even for a human so I won’t hold this against it (it did manage to find two reasonable sources). Not meh and actually a decent attempt.
What the bot told me it found, was validating my existing bias so I didn’t keep pushing. I also didn’t go and verify by myself.
Trying variations on earlier prompts
I tried again to get the bot to review my code for errors. As was becoming the norm, at my first prompt I got only generic answers; when prompted again the bot had apparently forgotten all about my code; and when I reminded it that it did have access to my code, then it finally obliged. Its answer was predictably full of extensive praise about each thing that I did correctly (in fact I was surprised that there were no errors, but given that, the bot’s response was predictable). I’d have been fine with a short list of common errors it checked for and didn’t find, but ok.
Then I tried to have the bot refactor my code in a few ways. I wanted to see if it could suggest code to send the data over the mesh in one format while writing on Serial in another format. I already knew it was bad at rewriting existing code.
That was a fiesta of all the advice it had given me this far, and it applied the bad variant every time. It knew by then of my preference for the “guard” style but ignored it; it made several copies of each scalar; it didn’t rewrite my code at all, instead using some made up example; it kept using functions that it had warned me against.
Spite
I tried trolling the bot by suggesting deep sleep again, but it can’t do sarcasm and it tried its best to give me points any time it could, so it didn’t contradict me and praised me again for my insight.
I also tried to see if a pointed question with no actual consequence would get flagged but alas, the bot helpfully (not!) provided a pro/con list to help me decide for myself, not realizing how futile the decision was. This one wasn’t about sarcasm, it was about comparing two identical options and telling me this in no uncertain terms.
Brainstorming
This is how I generally find this kind of bot useful: tossing around ideas, having it formulate them in several ways to help me solidify my own understanding of them.
I still tried to get it to suggest code with the libraries I am using, but I stopped insisting on refactoring my actual code. I also gave up on fine tuning the code according to the code standards established earlier in the conversation. At this point I have zero faith in their Memories product before even trying it 😂
(Perhaps an error I made here was in not telling the bot to forget the first search results)
The bot did make some assertions that look like they can hold water and I’ll have to verify them.
At one point I failed to see a correct expression in the suggested code, and when I told the bot that it was missing that expression, it didn’t call me out on it. I had to read the new answer and figure out that it was the exact same suggestion a second time. That’s annoying.
Getting an idea that might be good
I eventually saw that when trying to reconnect to the WiFi, with gradual increments on the delay inbetween attempts, eventually we get to a delay that is so long that perhaps the microcontroller staying awake doesn’t make sense.
For example when I’m doing maintenance on the Raspberry Pi or on the root node, none of the sensor readings matter for a while. They can try reconnecting a few times on signal failure in case it’s just a blip or a reboot, but if they fail for too long, they should feel free to go in power saving mode and try again much later.
So I gave the bot a basic description of this algorithm and it provided code that we iterated on for a bit. At one point I gave up trying with words and I pasted code myself, which earned me deep praise again and a line by line explanation of how great my code was. Of course the bot was eager to improve on this anyway, but logic isn’t its forte…
Parting thoughts
Now I need to iterate on this on my own, since in my case the nodes with a sensor have no concept of my home WiFi and they can’t reach the rpi0, only the root node can do that. So that’s the only microcontroller that would save power in the event that my home WiFi went down. Big improvement! /s
As for the bot itself, while it got better at conversing since earlier versions, and it learned how to search the web, fetch content and parse it really fast, and it can produce code that sometimes works, it’s still much less useful than a coding partner of any amount of experience.
I can’t gauge its trustworthiness or reliability, even when it’s sharing the links that it scanned, without reading through the materials myself.
It still can’t keep its facts straight, although it has access to API documentation and/or actual source code, and it rarely knows when it doesn’t have enough information.
[1] https://chatgpt.com/share/670d8875-dc8c-8007-97b1-9938b82ed838
[2] https://github.com/GuillaumeRossolini/griotte
It’s still in that “parlor trick by a useful idiot” stage. While I think it has great promise, it’s still like at the 51% of the scale for me on whether it ultimately helps or slows down tackling a challenge. For now, we’re the ones who need to get better at using it and knowing when it has gone senile. That, too, is often a challenge.
I need to change the storage mechanism
I’m doing backups of my readings database from the Raspberry Pi Zero to my PC, and what used to take 4 MB of space early on (in September) now takes 60 MB
Obviously even with indexes, SQLite can’t be expected to read this from flash storage to present my graphs in a timely manner
I guess I can double store: once in a daily database I can use to present my graphs, so that SQLite only ever has to read from tiny databases, and a second time in a single database meant for larger analysis (off the rpi)
The migration will be painful 😂
There is a minor issue with my IoT setup, and that is with DNS queries: they are absolutely flooding my PiHole logs 😅
Apparently, every time my gateway ESP32 forwards any readings to the Pi for storage, a new query reaches my local DNS resolver (which happens to be the same Raspberry Pi Zero); the ESP32 doesn’t seem to cache entries
At a rate of 1 reading every 3 seconds for each node, and just under a dozen of those…
Trying to tail pihole logs is a lost endeavor at the moment
I guess one solution would be with a dedicated IoT WiFi and its own DNS resolver?
I just added documentation to the README for my DIY mesh network of air quality sensors
Comments are welcome! This is still just a personal project
https://github.com/GuillaumeRossolini/griotte
#iot #esp32 #esp8266 #RaspberryPi #arduino #mesh #AirQuality
Latest in my little project: build a separate network of the same, that I can ship to someone else as a plug and play thing
I had lots and lots of network issues so I reworked the code a lot, I added keepalive messages and reduced memory leaks as much as I knew how
Pictured is from yesterday’s database, 7 nodes and only one reboot so that’s great
There are still things that I don’t understand, for example there are spots that I can’t place a node in? Like why can I place nodes at the edges of a room but not in the middle? Or why not directly across the wall from a working node? I have one in my office and another in the next room, but I can’t place one in the hallway in between? So weird
[edit] query can be found at https://github.com/GuillaumeRossolini/griotte/blob/main/app/stats1.sql
The new network had been running nicely until Tuesday at about 8 PM (I’m CET, and that’s currently UTC+2), when everything stopped and I didn’t realize until Thursday at about noon
At that point 4 of the 7 nodes came back up on their own, one having kept running the entire time (just disconnected), and the other three rebooted for an unknown reason
The last three nodes didn’t come back, so I forced a reboot yesterday
I guess the ESP32, which acts as a bridge between the mesh and the RaspberryPi, was overheating and it took a nap for two days?
I added heat sinks on the ESP32 and the Raspberry Pi Zero and, despite warmer temperatures on Thursday & Friday, there hasn’t been another issue (yet)
@jdlh specifically for COVID-19, you may want to look into this
https://youtu.be/U4a_kJkVUis
Install this with a basic air flow system, and you should be good to host events
(Of course I am no expert myself, so verify my claims)
@Zamfr it’s exactly your first thought, yes
(And thanks!)
@Zamfr I don’t know what a hub and spokes net is
Is that a topology where hubs segment the network?
Guillaume Rossolini
Morgan Davis
Jim DeLaHunt
Marius