Dualtronic

Blog

Websites for humans

Lately the web has been under a relentless onslaught of scrapers and other bots. Every site owner out there complains about the struggle to keep servers up in the face of what amounts to malicious activity. Been there myself; it's no fun. Besides, I'll be the first to say: generative AI? Kill it with fire!

But if we ruin the web for human beings in an attempt to keep LLMs out of our sites, the bad guys win.

Take Anubis for example: the site protector with a fierce name and cute anime mascot. It's usually fine... except my old tablet struggles to solve even a moderate challenge, thus effectively keeping me out of websites unless I happen to be at my PC. (My phone, despite being much newer and faster, does little better at this.)

This isn't even new. Beginning roughly two decades ago, everyone was waking up to the fact that search engines now ruled the web. Optimization became the word of the day. Hide your .php extensions! Add a sitemap! Haven't you heard search engines have stopped using the meta keywords tag?

I was guilty of the last one, and more besides: using canonical links; avoiding splash pages; flattening site structure and so on.

And you know what? All of it was wrong. Never mind that the big search engines don't even send you much traffic anymore; who cares about their absurd bureaucratic rules anyway? I'm a human being, writing for people.

• Tools like Shaarli also use page keywords.
• Mirroring pages across websites increases their reach.
• If your bot is confused by index.php at the end of an URL, maybe make it smarter.

No, if anything it's my readers I need to help. And you know what can give their browsers a hand? Stuff like:

• using semantic markup
• using alt text for images
• having autodiscovery for RSS feeds
• making fluid layouts that scale down to mobile devices
• adding dark mode

The only concession I still do to machines is adding OpenGraph markup for link preview cards. And even there, smarter apps like Mastodon have learned to fall back on the good old title and meta description elements.

Be right back, adding splash pages to more of my websites. Just for fun.