What the metrics remember: almost two decades of technical writing on ToughDev
I have been writing on ToughDev since 2008. At first the site was mostly a place to keep notes: old drivers, broken software, strange hardware behaviour, and fixes that I did not want to rediscover twice. After almost two decades, I started to wonder whether the writing itself had changed.
The machines are still there: old PCs, microcontrollers, radios, cameras, emulators, operating systems that probably should have been left alone. But perhaps the way I wrote about them had changed. Maybe the early posts were only technical notes, while the later posts kept more of the experiment around the result. So I decided to scrape my own blog and measure it.
Crawling my own archive
The experiment was simple enough. I crawled the public ToughDev article pages, saved the HTML locally, extracted the article text, split the text into paragraphs, and kept enough metadata to trace every result back to the original post. Each paragraph was also hashed, so later classification results could be joined back to the source paragraph without relying on fragile line numbers or ordering.
The final dataset looked like this:
- **articles_scanned**: 357 - **total_words_scanned**: 323028 - **candidate_first_person_paragraphs**: 2165 - **ai_personal_experience_paragraphs**: 543 - **parse_errors**: 0 - **personal_experience_rate_among_candidates_percent**: 25.08 - **year_range**: 2008-2026 - **median_article_words_overall**: 669.0 - **median_literary_creativity_score**: 35.218 - **median_technical_density_score**: 100.0
This is not a huge corpus by search-engine standards, but it is large enough for a personal technical blog. More importantly, it is my own archive, so I can usually tell when the output makes sense and when the script is simply fooling itself.
First obvious change: article length
In 2008, the median article was only 342 words. By 2025, the median article had grown to 1,471 words. That matches my memory of the site. The early posts were often short notes: I had a problem, found a workaround, and wrote it down. Later posts became closer to project logs. I started keeping more of the failed attempts, false assumptions, hardware quirks, and small discoveries along the way.
This does not mean every newer article is better. Some old short posts are still useful. Some long posts probably should have been shorter. But the shape of the writing changed. The blog became less like a notebook of fixes and more like a log of experiments.
Finding personal experience
A paragraph containing the word “I” is not automatically personal experience. For example, “I used this command” is just instruction. But “I tried this, it failed, and after several attempts I found the reason” is different. That kind of paragraph uses experience as evidence.
For the first pass, I searched for first-person candidate paragraphs. Then I used a local model to classify them into rough categories. Out of 2,165 first-person candidate paragraphs, 543 were classified as actual personal-experience paragraphs.
I expected project history or nostalgia to dominate. Instead, debugging was the largest category by far, which is expected. Most ToughDev posts are not “I built this and it worked.” They are usually “I thought this would work, it did not, and here is the trail of assumptions I had to remove one by one.”
In 2008, the classifier found only 0.148 personal-experience paragraphs per 1,000 words. By 2025, this increased to 3.411. In 2026, it reached 4.319, although 2026 has fewer posts and should not be over-interpreted.
This matches my experience writing the blog. Early articles are merely technical notes; later ones start to embed memory, frustration, failed assumptions, and the small reasons why a project was worth doing in the first place, explaining the rise in the personal experience paragraph count over the years.
Narrative technical prose: a rough score
Initially I assigned what I called a literary score to each paragraph, reflecting how creative or literary the paragraph sounds. Perhaps what was a bit too ambitious. The script does not know what literature is. It only looks for measurable signals: narrative markers, sensory words, figurative phrases, quoted speech, punctuation variety, and lower pure-code density.
Sometimes this works well. At other times, it simply finds a troubleshooting paragraph with a beginning, middle and end and give that a very high literary score. Among other things, the algorithm struggled because the score calculation was only done on articles published within the same year. The ranking would not work well if all articles published that year were highly technical. For example, the following Google Play troubleshooting paragraphs scored high (87.258) because it has a story shape: confusion, failed attempts, a clue, and finally a workaround, without being literary, by any stretch of the word:
A search on Google for these errors returned no conclusive results. Some replies suspected that Google Play analyzes the APK looking for design elements specific to tablets (e.g. looking for layout folder with names layout-sw600dp, layout-sw600dp-land, layout-sw720dp, layout-sw720dp-lan, etc. or looking for an XML layout with large screen width) while others say that Google Play is simply analyzing the screenshots I uploaded to see if it looked like a tablet app, not a phone app running on tablet with huge unused white space lying around. Well, if it’s indeed analyzing the screenshot, is there a way to make it think that my screenshots are tablet-compliant? The answer is, surprisingly, to use the Device Art Generator from Google itself and drag your phone app screenshot to the tool, selecting the Nexus 9 which has tablet resolution:
Where the literary score actually worked
The literary score did work quite well for other articles. The best examples came from the Pocket 386 article, with a score of 100.0. The subject was still technical: AdLib, Covox, TEMU, Tandy three-voice audio. But the reason for caring was no longer only technical. It was about whether the machine sounded the way I remembered it.
With this, MUSIC.PDM was finally able to play the sample songs it came with. During playback, the software just said “press any key to cancel playing” without any visual advancement of the score or indication of the currently sounding note. From a modern perspective – human-computer interaction, user experience design or ergonomics, whatever they call it – the lack of ongoing feedback was a clear limitation. Yet somehow I did find joy in mucking with MUSIC.PDM, learning how to follow the music aurally rather than visually, internalizing rhythm and contours by ear. Despite, or perhaps because of its simplicity, that little music composer eventually became my companion, and I would use it every day after school to compose my favourite childhood melodies.
The next paragraph, with a score of 98.561, also deserves its place:
Upon closer listening I realized that the articulated sound effect of each note was gone, or at least much less obvious on TEMU. This made the audio harsh, more like three PC speakers beeping in unison, unlike a true Tandy 1000 where each note has an attack and decay reminiscent of struck instruments. Back in the day, Tandy did everything they could to make the audio sound good within the severe constraints of the three-voice hardware. The speaker itself was physically cushioned, lending the sound a softer, warmer character that smoothed out harsh edges and masked many of the system’s limitations. The cheap Tecsun radio I was using as a speaker perhaps did not have any of this, producing audio that was inferior to what I remembered from the original machine.
This paragraph, written in memory of my grandfather, is probably the most emotional of all. It could perhaps make an MFA professor smile (on a good day). Yet its literary score is only 83.018, high but not as high as the rest. Obviously none of that would matter, as not everything in life can simply be measured by a soulless algorithm. Not even in this day and age of LLM and AI.
I went through the SNG song files in Deskmate directory, all of which have not been opened in years. I played each and every song, listening to the melodies once again. The music was playing, the notes were sounding, the three FM channels were unmistakably present. Still, I knew something wasn’t right. It just didn’t sound the way I remembered it on my Tandy 1000. To this day, I still vividly remember the warm sound, the soothing tone, and the lulling rhythm coming from that machine. Just like how I remember my grandfather’s voice when he passed me the Tandy. “I hope you will build a career on this”, he said. I continued to use the machine well into high school, and eventually did build a career in software development, just as what my grandpa had hoped for
In any case, these are the paragraphs I would have wanted the script to find. Not because they are technically difficult, but because the technical detail and the memory are hard to separate.
What hasn’t changed
The blog did not become less technical. The overall median technical-density score was still very high. Many years had a median technical-density score of 100.0. That sounds extreme, but it makes sense for a blog full of drivers, chips, file formats, commands, emulators, and hardware part numbers.
Obviously I could have tweaked the algorithm such that there would not be so many articles with a full technical score of 100. I didn’t bother with that. After all, a metric is just a metric. What perhaps matters more is the purpose behind the metrics. The objective of the entire exercise was just to see how my writing has improved over the years, while still maintaining the original spririt of ToughDev. Describing the solution to a technical problem, the circumstances behind it, and the enthusiasm I got when the outcome was what I wanted. That objective has no doubt been achieved.
From crude scores to local LLMs
This is not the first time I tried to use code to find interesting text. Back in 2018, I wrote about my crude attempt at processing the 1TB Yahoo Geocities archive. I used C# and LINQ to extract text files and HTML files, remove obvious binary files, split text into paragraphs, and assign a rough mood score based on hand-made word lists. Cheerful words added points, depressing words subtracted points, and the final score helped me find paragraphs worth reading among millions of extracted paragraphs.
That was not really NLP in the modern sense. It was more like make-shift text analysis held together by regular expressions, dictionaries, keyword lists, and enough trial and error to make the output usable. The entire task took around a week to complete.
This time, I just let a local LLM do some of the harder classification work, such as deciding whether a first-person paragraph is actually personal experience, and whether that experience is about debugging, hardware, project history, reverse engineering, nostalgia, or something else. Instead of LINQ, the newer version uses Python, pandas, matplotlib, Ollama, and Qwen 2.5 3B. The whole project was completed in less than a day.
Source downloads
The scripts are not meant to be a universal literary detector. They are only a reproducible way to interrogate one long-running technical blog.
The full process has three stages:
- Crawl the public ToughDev article pages and save the HTML locally.
- Run a deterministic analysis pass to extract article metrics, candidate paragraphs, and yearly summaries.
- Run an AI-assisted classification pass on the candidate paragraphs, then generate tables and charts for inspection.
The deterministic pass extracts article-level and paragraph-level metrics such as word count, paragraph count, first-person candidates, technical-density score, narrative markers, sensory words, and rough narrative-prose score.
The AI-assisted pass was done locally through Ollama. I used Qwen 2.5 3B for the paragraph classification. The task was not to generate prose, but to label candidate paragraphs into simple categories such as debugging, project history, hardware, reverse engineering, nostalgia, or other.
The rough workflow looked like this:
# 1. Create and activate virtual environment python3 -m venv .venv source .venv/bin/activate # 2. Install Python dependencies pip install requests beautifulsoup4 readability-lxml pandas matplotlib tabulate # 3. Crawl ToughDev public article pages python3 scrape_toughdev.py \ --start-year 2008 \ --end-year 2026 \ --out data/toughdev_html # 4. Run deterministic analysis python3 analyze_toughdev.py \ --html-root data/toughdev_html/articles \ --out output/toughdev_audit # 5. Pull local model for Ollama ollama pull qwen2.5:3b # 6. Run AI-assisted paragraph classification ./analyze_toughdev_ollama.py # 7. Generate tables, excerpts and charts python3 study_toughdev_results.py --option 1 python3 study_toughdev_results.py --option 2 python3 study_toughdev_results.py --option 3 python3 study_toughdev_results.py --option 5 python3 study_toughdev_results.py --option 7 python3 study_toughdev_results.py --option 9 python3 study_toughdev_results.py --option 11 python3 study_toughdev_results.py --option 10
The main generated outputs are:
output/toughdev_audit/summary.json output/toughdev_audit/metrics_by_year.csv output/toughdev_audit/metrics_by_article.csv output/toughdev_audit/candidate_paragraphs.csv output/toughdev_audit/ai_classifications.csv output/toughdev_audit/paragraph_style_metrics.csv output/toughdev_audit/literary_metrics_by_year.csv output/toughdev_audit/blog_pack/*.md output/toughdev_audit/charts/*.png
The main scripts are:
scrape_toughdev.py analyze_toughdev.py analyze_toughdev_ollama.py study_toughdev_results.py
The paragraph extraction is not perfect. Some boilerplate, quoted text, rating text or sharing text can still leak into the extracted output. The narrative score is only a proxy, rewarding measurable signals, not literary quality. The AI classification should be treated as an estimate rather than ground truth, since I did not manually label a validation set. The metric is still much more reliable and useful than the mood score I measured back in 2018 but is in no way a perfect indicator of writing quality. A score can find a paragraph. It cannot decide whether the paragraph matters. That part still belongs to the author.
Download all the above scripts here.
See also
Making good sense of the 1TB Yahoo Geocities data archive
