@lyse@lyse.isobeef.org … I sure hope that they generate these files from the general terminfo database instead of maintaining their own DB. 😳

@movq@www.uninformativ.de I noticed that your feed’s last modification timestamp was missing in my database. I cannot tell for certain, but I think it did work before. Turns out, your httpd now sends the Last-Modified with UTC instead of GMT. Current example:

Sat, 03 Jan 2026 06:50:20 UTC

I’m not a fan of this timestamp format at all, but according to the HTTP specification, HTTP-date must always use GMT for a timezone, nothing else: https://httpwg.org/specs/rfc9110.html#http.date

At around 19 seconds in the video, you can see some minor graphical glitches.

Text mode applications in Unix terminals are such a mess. It’s a miracle that this works at all.

In the old DOS days, you could get text (and colors) on the screen just by writing to memory, because the VGA memory was mapped to a fixed address. We don’t have that model anymore. To write a character to a certain position, you have to send an escape sequence to move the cursor to that position, then more escape sequences to set the color/attributes, then more escape sequences to get the cursor to where you actually want it. And then of course UTF-8 on top, i.e. you have no idea what the terminal will actually do when you send it a “🙂”.

Mouse events work by the terminal sending escape sequences to you (https://www.xfree86.org/current/ctlseqs.html#Mouse%20Tracking).

ncurses does an amazing job here. It’s fast (by having off-screen buffers and tracking changes, so it rarely has to actually send full screen updates to the terminal) and reliable and works across terminals. Without the terminfo database that keeps track of which terminal supports/requires which escape sequences, we’d be lost.

But gosh, what a mess this is under the hood … Makes you really miss memory mapped VGA and mouse drivers.

@prologic@twtxt.net In my opinion, the integrity isn’t lost. The same input data always result in the same output hash, no matter when you calculate the hashes. It’s true that a corrupt database contents yields to corrupt hashes, but then you have a whole bigger problem than just receiving different hashes. :-D

@zvava@twtxt.net By hashing definition, if you edit your message, it simply becomes a new message. It’s just not the same message anymore. At least from a technical point of view. As a human, personally I disagree, but that’s what I’m stuck with. There’s no reliable way to detect and “correct” for that.

Storing the hash in your database doesn’t prevent you from switching to another hashing implementation later on. As of now, message creation timestamps earlier than some magical point in time use twt hash v1, messages on or after that magical timestamp use twt hash v2. So, a message either has a v1 or a v2 hash, but not both. At least one of them is never meaningful.

Once you “upgrade” your database schema, you can check for stored messages from the future which should have been hashed using v2, but were actually v1-hashed and simply fix them.

If there will ever be another addressing scheme, you could reuse the existing hash column if it supersedes the v1/v2 hashes. Otherwise, a new column might be useful, or perhaps no column at all (looking at location-based addressing or how it was called). The old v1/v2 hashes are still needed for all past conversation trees.

In my opinion, always recalculating the hashes is a big waste of time and energy. But if it serves you well, then go for it.

@zvava@twtxt.net The problem you now then is you lose integrity of the message content if you compute the hashes at runtime rather than on the way in. So if your message content or database becomes corrupt in any way, so do your hashes.

@lyse@lyse.isobeef.org while caching those is a good idea the problem is baking data that can be calculated into the database instead of some cache, because post hashes are not fixed and change for every post edit. you can always easily look up other twts by hash with a cached lookup table, but now you’re not locked into them so supporting hashv2 or other hash variants or any other solution becomes far easier

@zvava@twtxt.net I might misunderstand what you wrote, but only hashing the message once and storing the hash together with the message in the database seems a way better approch to me. It’s fixed and doesn’t change, so there’s no need to recompute it during runtime over and over and over again. You just have it. And can easily look up other messages by hash.

very good blog post that reminded me why it’s taking so long to ship bbycll — previously i had computed the hashes of every post before storing them in the database, after realizing it’s a much better idea to compute the hashes during runtime and only store the post content & timestamp i’m now having to rewrite every function that reads & writes data. i hope the reason as to why i lost motivation is obvious — thankfully i caught it early enough so that once i’m done rewriting just those functions i should™ be able to finalize 1.0-rc with little hassle

⇒ the cardinal sin of software architecture: the unnecessary distribution, replication, or restructuring of state, both in space and time.

I had a looksie (just to be sure) at the database, and they were thankfully legit test events. But this did spark/trigger me to make sure I have some form of anti-spam measures in place. So I added some per-event / per-rsvp rate-limiting and honeypot(s).

i desperately want to simply deploy a bbycll instance but i need to change the entire database schema for the Nth time, i haven’t been working on it much recently as this back and forth w the backend (which you don’t expect from a spec as simple as twtxt) is really demotivating, as well as life and stuff getting in the way

perhaps i just need a nice shower and four coffees, midnight is when i am most productive and the hour is approaching..

Hopefully I can muster up the energy to start this new project:

Put up lots of thermometers and hygrometers in the apartment, have them report their readings wireless to a database.

I suspect that I’ll have to “build” these myself, because ready-to-use kits most like require some sort of cloud service. Dunno, haven’t checked yet.

@prologic@twtxt.net I too, self-host various services on a VPS (and considering buying a mini PC to keep at home instead).

I use most of it as a hosting platform for personal use only and as a remote development environment (I do share a couple of tools with a friend though).

But given the costant risks of DDoS, hacking, bots, etc. I keep any of my public facing resources purely static and on separate hosting providers (without lock-ins of course).

Lately, I began using homebrew PWAs with CouchDB as a sync database, this way I get a fantastic local-first experience and also have total control of my data, that also sync in a locally hosted backup instance in real-time.

Also, I was already aware of Salty.im, but what I’m thinking is a more feature complete solution that even my family can use quickly, Delta.chat with the new chatmail provider (self-hostable) might be the solution for my needs.

But I’m still thinking if it’s worth the trouble. I might just drop everything and only use safe channels to speak with them (free 24/7 family tech-support is easy to manage 😆).

Also, I’ll be waiting for the day you’ll share with us your story, I’m pretty curious about it!

@alexonit@twtxt.alessandrocutolo.it My problem is I don’t see a world where we don’t employ some form of cryptography to use as keys for threads in databases and other such things honestly. I’m not going to use url#timestamp as keys.

@lyse@lyse.isobeef.org @prologic@twtxt.net Can’t we find a middle ground and support both?

The thread is defined by two parts:

1. The hash
   
2. The subject
   

The client/pod generate the hash and index it in it’s database/cache, then it simply query the subject of other posts to find the related posts, right?

In my own client current implementation (using hashes), the only calculation is in the hash generation, the rest is a verbatim copy of the subject (minus the # character), if this is the common implemented approach then adding the location based one is somewhat simple.

function setPostIndex(post) {
    // Current hash approach
    const hash = createHash(post.url, post.timestamp, post.content);

    // New location approach
    const location = post.url + '#' + post.timestamp;

    // Unchanged (probably)
    const subject = post.subject;

    // Index them all
    addToIndex(hash, post);
    addToIndex(location, post);
    addToIndex(subject, post);
}

// Both should work if the index contains both versions
getThreadBySubject('#abcdef') => [post1, post2, post3]; // Hash
getThreadBySubject('https://example.com#2025-01-01T12:00:00') => [post1, post2, post3]; // Location

As I said before, the mention is already location based @<example https://example.com/twtxt.txt>, so I think we should keep that in consideration.

Of course this will lead to a bit of fragmentation (without merging the two) but I think this can make everyone happy.

Otherwise, the only other solution I can think of is a different approach where the value doesn’t matter, allowing to use anything as a reference (hash, location, git commit) for greater flexibility and freedom of implementation (this probably need the use of a fixed “header” for each post, but it can be seen as a separate extension).

I corrupted my SQLite test database with sed -i s/… $(find …). Clearly, I found too many files. That’s the signal to go to bed.

@bender@twtxt.net just a heads up im thinking of rewriting the database schema with hash v2 in mind >.<

@kat@yarn.girlonthemoon.xyz @kat@yarn.girlonthemoon.xyz Pretty sure I have many more mentions in the database than the one and only one I see hmmm 🤔 – I’ll have a look at the code when I can and the SQL query it’s using

@bender@twtxt.net i also recently discovered there was a bug causing new users to initialize wrong leading to their posts not being saved :p ..and made breaking changes to how the config and database files are stored so um, make sure to clear your local tree before updating!

Chances are the database bought wasn’t cheap at all and was aold by some scam company that probably ripped them from six figures or more for a database that’s full of rubbish. 🤣

Now that’s interesting. Some of these bots start crawling at URLs like this:

https://uninformativ.de/projects/lariza/NetTracer-Scenes/GPUTracer/multipass/xlonitor/http-collect/getpw

That is obviously completely wrong. But I can explain it. Some years ago, I screwed up my nginx rewrite rules, and that’s how these broken URLs came to be.

It all redirects to /git now, which is why that endpoint sees so much traffic lately.

But what does that mean? Why do they start there? I can only speculate that this company bought an old database of web links and they use that to start crawling. And it was probably a cheap one, because these redirects have been fixed for quite a long time now.

linode’s having a major outage (ongoing as of writing, over 24 hours in) and my friend runs a site i help out with on one of their servers. we didn’t have recent backups so i got really anxious about possible severe data loss considering the situation with linode doesn’t look great (it seems like a really bad incident).

…anyway the server magically came back online and i got backups of the whole application and database, i’m so relieved :‘)

@kat@yarn.girlonthemoon.xyz yes, both the newsletter and the podcast, from time to time.

@prologic@twtxt.net I was not expecting much, but since the list of restaurants near company buildings, was hard coded into it, I did expect it to at least copy the menu text, from the websites, in its database. Ironically, the only restaurant where it got something right, is the only one, where the websites has the text as a transparent PNG, the AI has to convert to text.

Wikipedia: Database Download
Comments ⌘ Read more

I asked ChatGPT what it knows about Twtxt 😂 And surprisingly it’s rather accurate:

Twtxt is a minimalist, decentralized microblogging format introduced by John Downey in 2016. It uses plain text files served over HTTP—no accounts, databases, or APIs.
In 2020, James Mills (@prologic@twtxt.net) launched Yarn.social, an extended, federated implementation with user discovery, threads, mentions, and a full web UI.
Both share the same .twtxt.txt format but differ in complexity and social features.

Democrats: Doge building a ‘master database’ of Americans’ sensitive information
Comments ⌘ Read more

@bender@twtxt.net Exactly. I suspect it was because of sqlitebrowser also accessing the database in parallel to debug the original issue.

So far, I have not found the exact reason why some replies don’t show up. When I do not filter for unread messages and show all, though, I actually see them. So, there’s that.

I just noticed that my unread messages counter was off by quite a bit. It showed 8, but I only saw one unread message. Even after restarting my client, which recalculates the number of unread messages, it remained at eight. Weird. Looking in the database revealed that this is indeed correct.

Apparently, my query to build up the message tree must be incorrect. It somehow misses seven messages. They all are orphaned, maybe that’s a clue. However, generating missing root messages (and thereby including the replies) typically works just fine. Hmm.

@movq@www.uninformativ.de json and database put together sounds terrifying. i must try jenny

jenny really isn’t well equipped to handle edits of my own twts.

For example, in 2021, this change got introduced:

https://www.uninformativ.de/git/jenny/commit/6b5b25a542c2dd46c002ec5a422137275febc5a1.html

This means that jenny will always ignore my own edits unless I also manually edit its internal “json database”. Annoying.

That change was requested by a user who had the habit of deleting twts or moving them to another mailbox or something. I think that person is long gone and I might revert that change. 🤔

Manga Guide To Databases (2004) [pdf]
Comments ⌘ Read more

A threat model for opposing authoritarianism
A decade ago, I published a book on privacy “Dragnet Nation: A Quest for Privacy, Security, and Freedom in a World of Relentless Surveillance.” In the book, and since then, in articles and speeches, I have been dispensing advice to people on how to protect their privacy. But my advice did not envision the moment we are in – where the government would collaborate with a tech CEO to strip-mine all of our data from government databases and use i … ⌘ Read more

European Union Vulnerability Database (EUVD)
Comments ⌘ Read more

Windows Recall returns, and its companion feature does not keep data on-device
Remember Windows Recall, the Windows feature that would take a screenshot of your desktop every three seconds, stored them in a database, and then let you search through them at later dates? The feature has been hobbled by implementation problems, security issues, and privacy troubles, and has been released in preview and pulled since its original unveiling. Well, it’s back in … ⌘ Read more

@prologic@twtxt.net is it twice on database, or simply rendering twice? If you manually expunge it, will it affect the yarn?

DIY Synths Database
Comments ⌘ Read more

@xuu@txt.sour.is Wow, that’s a giant graveyard. In my new database I have 16,428 messages as of now. Archive feed support is not yet available, so it’s just the sum of all the 36 main feeds.

Thanks, @movq@www.uninformativ.de!

My backing SQLite database with indices is 8.7 MiB in size right now.

The twtxt cache is 7.6 MiB, it uses Python’s pickle module. And next to it there is a 16.0 MiB second database with all the read statuses for the old tt. Wow, super inefficient, it shouldn’t contain anything else, it’s a giant, pickled {"$hash": {"read": True/False}, …}. What the heck, why is it so big?! O_o

A collection of postgreSQL patterns that you can use in other databases
https://mccue.dev/pages/3-11-25-life-altering-postgresql-patterns
#postgresql #databases

(Back in tt.) Well, it kinda worked. At least appending to the file. But my cache database got screwed up. I do not yet support replies, so the subject and and root hash columns have not been set at all, resulting in a message that is just not shown at all. I gotta do something about that next. The good thing is, though, after simply fixing the two columns the message appeared on screen.

wahhh i wanna work towards my dream of offering pay as you can web hosting (static & dynamic) but i don’t know how!!!!! i keep drifting towards hosting panels but i don’t exactly have fresh linux servers for those nor do i like the level of access they require. so i’m like ok i can do the static site part with SFTP chroot jails and a front-end like filebrowser or something…. but then what about the dynamic sites!!!!!!! UGH

granted i doubt i’d get much interest in dynamic sites but i’d like to do this old school where i can offer people isolated mySQL databases or something for some project (i’m thinking PHP based fanlistings), which means i could do it the old school way of… people ask me to run it and i do it for them. but i kind of want to let people have access to be able to do it themselves just short of giving them SSH access which isn’t happening

@andros@twtxt.andros.dev If something fits in a CSV file, it typically doesn’t require a database. I agree with that. Depending on the application, more complicated queries might benefit from a database, though. I don’t know awk very well, but I could imagine that grep, sed and cut reach their CSV processing limits rather quickly when you have to deal with escaped (multiline) fields.

I only very rarely have to deal with CSV files or databases in my day to day life. Maybe, these classic Unix tools offer some tricks I’m not aware of. When I have some more complicated CSV input, I generally reach for Python.

pls elaborate on a ‘p2p database’, ‘all story’ and ‘Registries’.

My first thought takes me to something like secure-scuttlebutt which it’s painful to sync data using clients, and too slow compared to downloading a text file.

Also I’d like for twtxt to avoid becoming an ActivityPub. Works well but it’s uses too many resources IMO.
https://kingant.net/2025/02/mastodon-the-cost-of-running-my-own-server/

I’m defending being able to self-host your Web client (like you’d do with a Wordpress, twtxt is a micrologging, at the end), instead of federated instances, so in a first thought I’d say Registries have many disadvantages being the first one that someone has to maintain them active.

What does the #twtxt community think about having a p2p database to store all history? This will be managed by Registries.

@prologic@twtxt.net We often turn to a database when we can use a plain text file, such as a CSV. With sed or awk, you can run simple queries without using a database.
Did I get the context right? 😀

The other day, after a discussion online, we came to the conclusion that using awk+sed+tr could replace much of the development that requires a database. However, using SQLite to have a SQL syntax isn’t a bad idea either. What do you think?

I’m continuing my tt rewrite in Go and quickly implemented a stack widget for tview. The builtin Pages is similar but way too complicated for my use case. I would have to specify a mandatory name and some additional options for each page. Also, it allows me to randomly jump around between pages using names, but only gives me direct access the first, however, not the last page. Weird. I don’t wanna remember names. All I really need is a classic stack. You open a new fullscreen dialog and maybe another one on top of that. Closing the upper most brings you back to the previous one and so on.

The very first dialog I added is viewing the raw message text. Unlike in @arne@uplegger.eu’s TwtxtReader, I’m not able to include the original timestamp, though. I don’t have it in its original form in the database. :-/

Next up is a URL view.

I think it is not easy to implement, you need a database. Timeline is an elegant solution: read and sort.

FINALLY!! Got #Caddy server up and running and got rid of nginx proxy manager and Mysql database containers 🥳🥳🥳

What is clean architecture? That’s a good question.

You think of a pattern for ordering code with good decisions isolating technologies (you can change the web framework or database without break the business logic), easy to test (you only test interfaces and use cases), sharing code between frameworks (entities and use cases), scalability, modulations and standardizing names. Clean architecture is not perfect, it has a learning curve and some abstraction in each technology. You can even find rejection with yours colleagues.
I have a good article on this topic.
https://programadorwebvalencia.com/implementando-arquitectura-limpia-en-python/
#python