There it is... 44 seconds for 4096 KiB of data, 11s per MiB.

After trying different settings for the sd card ISP connection and baudrate, and verifying that the sd card is written in 512 blocks, this is the best I can come to without writing my own class.

Given that the time needed for writing the file is now only 10% compared to the read process I am quite happy about this number ( used to be 120%-130% when using the on board flash ).

of course I cannot make the 3 shift registers, onboard flash and QSPI work any faster than their datasheets, but I want to make it an option to use that if someone does not want to add an sd card, and I will also leave the 230400 baudrate UART option around if you really don't care and just want to stream the data out using serial.

Next up, the LCD, rotary encoder, buttons, writing SRAM and debug firmware... I got this idea to apply patches on the fly and want to check out how it could be done.

That said, I will release this soon-ish, after I clean up and standardize all the time saving methods on all the files, It is now in a good enough state to receive outside feedback on further improvements.

I also have none of the interesting cartridges to go the next challenge, only some DSP-1 ones.

I wanted to use the PIO to generate a second/third clock and even emulate the CIC chip in the future for those special co-processors... but I need some games for that to happen 😆

Interesting....

with very minor changes, particularly forcing higher baudrate and adding some gc threholds and collection to cover for the buffers, the sd class can now push much faster the data to the sd card, almost reaching the baseline limit....

Definitely worth looking closer and making maybe a new, custom, sd class.... ? with 11.5s per MiB I am finally getting happier on these times though.

I also wanted to look into making the debug firmware anyway...

need to organize my thoughts, it is late as well...

ok, adding a micro sdcard makes sense, but marginally as it is now...

I am using https://github.com/micropython/micropython-lib/blob/master/micropython/drivers/storage/sdcard/sdcard.py as class and got a speedup of about 15 seconds, bringing the read time to ~19s per MiB, or 54KiB/s in avg, that is some pretty bad write times though...

If I just count the baseline read time ( no file writes ), that is 10s Per MiB, which is definitely a very good point to reach.

Looking at the SDcard class, I feel there could be some room for improvement on speed and will look into that next. I will add an sdcard though on the PCB footprint as there are Cartridges out there that cannot fit on a 16MiB flash alongside codebase.

Will also go back to the oscilloscope in order to calculate the actual max write speed. obviously the avg is not the one I hit a ceiling at, most likely is is closer to 100KiB/s or so.

Implemented checksum check and filesystem available size checks.

Viper is pretty good at doing these math, but I also took advantage that I have the wait time between the setting of the address lines and the cart presenting the data.

Basically I am adding the sum for the checksum instead of the dead time of the wait loop, this means that I can reduce the wait loop time by the amount of time viper needs to add the numbers and I don't get a big hit on performance, that said, viper is very fast and I still need to have some wait loop in place.

The timings on these speeds are tricky and it shows. I am getting a hit of about 1 second per MiB when calculating checksum is enabled, but it is worth it, downloading a file to check if dumped correctly takes about 10s per MiB.

I tried expanding the bytearray to read two banks per loop, but that did not seem to improve the time needed for a full dump. Looking into the oscilloscope it showed why... writing data in the flash storage is taking always a specific amount of time, and it is analogous to the bytearray size ( ofc ).

So... next step is to see how I can design something that will make the writes to the flash chip async or buffered ? If I could do that then I can half the amount of time again to less than 10s per MiB as currently the time needed to write a 64KiB bytearray to flash is ~120ms more than the actual flash read from the cartridge itself.

I wonder if #micropython has any way for me to do that... especially in a viper function... apparently the micropython forum is down for some time and only returns a 404 so I cannot see any previous answers on this matter 😐

Maybe adding an sdcard instead of the flash would help ? I wonder...

ok, I was doing something embarrassingly stupid when allocating the full bank bytearray... ( using append() 😅 )

I am now on 90 seconds for a HiROM 4MiB cartridge, and 22 seconds on a 1MiB LoROM

That is definitely much better and I can live with this, but will look for even more optimizations on bank loops, they are the only thing remaining to improve without removing a shift register...

After implementing the read and write to flash to happen by a full bank and not per Byte ( and refactored the pio to use one less instruction )... 15 seconds gain on the 4 MiB roms, pretty much what I expected. The gain is higher on LoROMs but comparable ( 32KiB vs 64KiB Banks )

Now.. to start writing everything in Viper and cleaning things around, also, I need to write a checksum validator step as currently I keep having to download the file and open it on an emulator in order to check that the read worked...

Will have to analyze again the rates on an oscilloscope, the main bottleneck is the setting of 24 bits on the 74HC595 shift registers and the wait for the data line to be set from the cartridge.

The shift registers are already over their datasheet limits. With a 62.5 MHz PIO freq, I am already driving them on a 31.25 MHz clock as the clock on and off states are right after each other ( verified with oscilloscope ), so the next step is to parallelize somehow the wait for the data line to happen with the setting of addresses on the shift registers, a definitely interesting challenge to solve.

That said, I am approaching the theoretical limits on the speed I can get with the current set up... I have 14 pins free on the pico and I can afford to use 8 for the bank setting while also reducing the cost and decreasing the address setting time by 33%... will see... as I also wanted to have an lcd screen and sdcard in the future.

Overall I should be happy with this time, but knowing that I am not even close to the cartridge specs makes me want to push much harder...

I guess I am closing the time for a new PCB design with fixes, sdcard, lcd and push buttons and one less shift register, or maybe even replacing them with an I2C based one.. more math I guess 😇