CCTVs and SD card life expectancy

To most users, flash memory wear out is typically not a concern, even more so when they have to ask about it. On my daily workstation running on a 2TB Western Digital SSD, the wear-out indicator is still at around 0.68% after almost two years of usage with almost 14TB of data being written to the devices. At this rate the SSD will probably be outdated long before the wear-out indicator reaches a critical level:

Obviously, the above assumption is not true for all users, especially those who need to write several gigabytes of data to disk daily. This turns out to be the case in my CCTV setups, in which a central Raspberry Pi running on a 128GB Sandisk SD card is tasked with backing up all recent CCTV recordings for the past few days, with older recordings being deleted once the SD card becomes full. The system had been working well for around a year and half until one day, all of a sudden, the CCTV backup process started to generate a lot of errors. The card could still be mounted in Ubuntu as well as Windows, and a quick check showed that roughly 15TB of data has been written to the card:

dumpe2fs /dev/mmcblk0p2 | grep Lifetime >> $OF
Lifetime writes: 15 TB

Small files could still be written to the card with no data loss. I was also able to format the card and create a single partition on it with the help of gparted. However, Windows froze after a short while when I attempted to copy a 2.33GB file to the card:

I guess the flash controller could no longer find enough spare blocks to hold the large file, and the copying process just froze. Reading from the card was still perfectly fine, and I was able to copy the contents to a brand new 128GB card of the same brand to get the system up and running again within a couple of hours. The fast wear-out is probably because of data being constantly written to the card at a rate of around 30GB daily. For a 128GB SD card to fail at around 15TB of written data, every memory cells would have been written to and erased just over 100 times. So what about the typical 10,000, 100,000 erase cycles or even 1,000,000 erase cycles of flash memory cells you often read about? I will leave it as an exercise for the reader to think about, but I suspect that the cheapest possible memory cells are used to manufacture SD cards, as compared to USB or SSD devices, and the number of supported erase cycles might be something like 500 or 1000. The flash controller is simply designed to return failure after a certain number of erase cycles (which is around 100 in this case), even though the memory cells might still be perfectly fine.

I have another 256GB SD card running in a similar setup, which is still going strong after 12TB of data has been written to it. Will this card fail at 15TB, 30TB or some other figures? Only time will tell.

During my tests, SD cards 32GB or less from major brands continue to work well with no read/write issues even after several hundred GB of data has been written to the cards with no indications of any imminent failure. However, there is an exception to every rule and sure enough, one of the cards failed at just 45GB – reads were still ok with no corrupted data but writes were simply ignored with no reported errors, unlike the 128GB SD card above. The card would revert to its original state before the writes after a short while.  For this case, as the failure came much sooner than expected (at 45GB of written data for a 16GB SD card), I suspect the issues is with the flash controller, rather than anything to do with flash memory wear-out.  Other 16GB cards still work well after several hundred GB of written data. Alternatively, another explanation is that the card was second-hand and had been used previously, which cannot be ruled out as I forgot where this card was originally purchased from.

Putting any concerns of memory wear-out aside, I would like to add that heat is the enemy of most electronics, and SD cards or USB thumb drives expected to be used in any data logger systems will need to be sufficiently cooled or the media might die of heat long before wear-out becomes an issue. Regardless of which brands you use, always remember to perform periodic checks on the SD cards to ensure data integrity.  And as always, do have backups that can be easily stored when the need arises and you should be doing fine with your CCTV system