July 1st. 63 days from now to implement a backward-incompatible change, apparently not open to other ideas like replacing blake with SHA, or discussing implementation challenges for other languages and platforms.
Finally just closing #18, #19 and #20 without starting a proper discussion and ignoring a ‘micro consensus’ feels… not right.

I don’t know what to think rather than letting it rest (May will be busy here) and focus on other stuff in the future.

twt-hash-v2.md#implementation-timeline

I will be adding the code in for yarnd very soon™ for this change, with a if the date is >= 2025-07-01 then compute_new_hashes else compute_old_hashes

Finally I propose that we increase the Twt Hash length from 7 to 12 and use the first 12 characters of the base32 encoded blake2b hash. This will solve two problems, the fact that all hashes today either end in q or a (oops) 😅 And increasing the Twt Hash size will ensure that we never run into the chance of collision for ions to come. Chances of a 50% collision with 64 bits / 12 characters is roughly ~12.44B Twts. That ought to be enough! – I also propose that we modify all our clients and make this change from the 1st July 2025, which will be Yarn.social’s 5th birthday and 5 years since I started this whole project and endeavour! 😱 #Twtxt #Update

I had Chick-fil-A breakfast today (sausage, egg, and cheese biscuit, hash browns, coffee, and orange juice). Then at lunch my work place offered hot dogs. I had two (kosher, if that matters), plus a coke, a macadamia nuts cookie, and a small chocolate brownie.

So, here I am, at home, feeling hungry but guilty and refusing to eat anything else for the rest of the day. To top it off, I have only clocked 4,000 steps today (and I don’t feel like walking). I am going to hell, am I?

by commenting out DMs are you giving up on simplicity? See the Metadata extension holding the data inside comments, as the client doesn’t need to show it inside the timeline.

I don’t think that commenting out DMs as we are doing for metadata is giving up on simplicity (it’s a feature already), and it helps to hide unwanted DMs to clients that will take months to add it’s support to something named… an extension.

For some other extensions in https://twtxt.dev/extensions.html (for example the reply-to hash #abcdfeg or the mention @ < example http://example.org/twtxt.txt >) is not a big deal. The twt is still understandable in plain text.
For DM, it’s only interesting for you if you are the recipient, otherwise you see an scrambled message like 1234567890abcdef=. Even if you see it, you’ll need some decryption to read it. I’ve said before that DMs shouldn’t be in the same section that the timeline as it’s confusing.

So my point stands, and as I’ve said before, we are discussing it as a community, so let’s see what other maintainers add to the convo.

After reading you, @eapl.me@eapl.me, I’ll tell you my point of view.
In my opinion, a feed does not have to be equivalent to a timeline. A timeline is a representation of the feed adapted to a user. You may not be interested in seeing other people’s threads or DMs. But perhaps they are interested in seeing mentions or DMs directed at them. It is important not to fall into the trap. With that clarification…
I insist, this is my point of view, it is not an absolute truth: I don’t think extensions should be respectful of customers who are no longer maintained.
We cannot have a system that is simple, backwards compatible and extensible all at the same time. We have to give up some of the 3 points. I would not like to give up simplicity because it will then make it harder to maintain the customers who do stay. Therefore, I think it is better to give up backwards compatibility and play with new formulas in the extensions. I don’t think it’s a good idea to make a hash keep so much load: a hashtag, a thread and also a DM.

yes it used be http:// only and to keep hashes from breaking i added # url = http://... and now we are stock with it due to the curret specs.

Hmmm there’s a bug somewhere in the way I’m ingesting archived feeds 🤔

sqlite> select * from twts where content like 'The web is such garbage these days%';
      hash = 37sjhla
  feed_url = https://twtxt.net/user/prologic/twtxt.txt/1
   content = The web is such garbage these days 😔 Or is it the garbage search engines? 🤔
   created = 2024-11-14T01:53:46Z
created_dt = 2024-11-14 01:53:46
   subject = #37sjhla
  mentions = []
      tags = []
     links = []
sqlite>

Some A hole has been trying to pull every single Twtxt feed that existed/still exists since forever. How do I know? Welp’ They’ve been querying my Timeline™ instance for all of it, every single twtxt file and twt Hash they can find. 😆🤦 It must have been going on for days and I have just noticed… + it’s all coming from the same ASN AS136907 HWCLOUDS-AS-AP HUAWEI CLOUDS

Thank you Huawei for the DDos you sons of Glitches!!!

@quark@ferengi.one No editing old Twts that are the root of a thread with replies in the ecosystem. Just results in a fork. Unless the client has an implementation that does not store Twts keyed by Hash.

Ha! I stand corrected, didn’t scrolled long enough. Indeed, it should be added (you will need an account on Mills’ Gitea), noted.

@eaplme@eapl.me you wrote:

“That PHP snippet could be merged into https://twtxt.dev/exts/twt-hash.html”

Why, though? AFAIK @andros@twtxt.andros.dev’s client is on Emacs, @lyse@lyse.isobeef.org’s is on Python (and Golang, for tt2), @movq@www.uninformativ.de’s is on Python, and @prologic@twtxt.net’s is on Golang. All the client creator needs to know is in the documentation already, coding language agnostic.

just a note that we are doing that on PHP: https://github.com/eapl-gemugami/twtxt-php/blob/master/docs/03-hash-extension.md#php-72

That PHP snippet could be merged into https://twtxt.dev/exts/twt-hash.html

@david@collantes.us @andros@twtxt.andros.dev The correct hash would be si4er3q. See https://twtxt.dev/exts/twt-hash.html, a timezone offset of +00:00 or -00:00 must be replaced by Z.

(That said, there’s a bug in jenny as well. It only replaces +00:00, not -00:00. 🤡)

@andros@twtxt.andros.dev the hash on @aelaraji@aelaraji.com’s last message (as I type this) is:

[si4er3q] [2025-04-16 22:49:11+00:00] [Am I tripping or `rsync` is actually THIS effing faster than `scp`!!? 🫨]

So, si4er3q

@prologic@twtxt.net @bender@twtxt.net

What is the hash of the last message from?: https://aelaraji.com/twtxt.txt

@bender@twtxt.net @aelaraji@aelaraji.com The client should ignore twts if it’s not compatible or not addressed to me. it’s a simple regex to add! It’s similar to Twt Hash Extension, should they be in another file? They are child messages, not flat twt. Not of course!

@prologic@twtxt.net interesting. What would happen on a hash collision? 🤔

@bender@twtxt.net It’s a bug in the UI for sure. The hash is the primary key.

@david@collantes.us Yeah, we’ve been debugging that a bit yesterday. Looks like the wrong input (sometimes) gets fed to the hash function → broken threads.

@movq@www.uninformativ.de @kat@yarn.girlonthemoon.xyz Heck yeah, that’s crazy! :-) Fingers crossed! (tt also agrees with the right™ hash)

The actual hash is fs7673q.

@prologic@twtxt.net that’s not what I see. The hash znf6csa cannot be found.

@prologic@twtxt.net There was no edit according to my Git history. 🤔 On my end, the hash is fs7673q and that’s also what kat used to reply.

Doesn’t look like it Hmmm

sqlite> select * from twts where content LIKE '%Linux installation%';
    hash = znf6csa
feed_url = https://www.uninformativ.de/twtxt.txt
 content = I wonder if my current Linux installation will actually make it to 20 years:

    $ head -n 1 /var/log/pacman.log
    [2011-07-07 11:19] installed filesystem (2011.04-1)

It’s not toooo far into the future.

It would be crazy … 20 years without reinstalling once … phew. 🥴
 created = 2025-04-07T19:59:51Z
 subject = (#znf6csa)
mentions = []
    tags = []
   links = []

@movq@www.uninformativ.de Apparently you wrote it :D The hash doesn’t lie? 🤣 https://twtxt.net/twt/znf6csa

@prologic@twtxt.net What happened here – did I edit my twt or is this hash wrong? 🥴

@prologic@twtxt.net Spring cleanup! That’s one way to encourage people to self-host their feeds. :-D

Since I’m only interested in the url metadata field for hashing, I do not keep any comments or metadata for that matter, just the messages themselves. The last time I fetched was probably some time yesterday evening (UTC+2). I cannot tell exactly, because the recorded last fetch timestamp has been overridden with today’s by now.

I dumped my new SQLite cache into: https://lyse.isobeef.org/tmp/backup.tar.gz This time maybe even correctly, if you’re lucky. I’m not entirely sure. It took me a few attempts (date and time were separated by space instead of T at first, I normalized offsets +00:00 to Z as yarnd does and converted newlines back to U+2028). At least now the simple cross check with the Twtxt Feed Validator does not yield any problems.

@andros@twtxt.andros.dev sha256 hash of twt in json. Look at converter script

Amazing! It is a good tool for reading feeds. What you used to calculate the hash?

Hello, i want to present my new revolution twtxt v3 format - twjson
That’s why you should use it:

1. It’s easy to to parse
   
2. It’s easy to read (in formatted mode :D)
   
3. It used actually \n for newlines, you don’t need unprintable symbols
   
4. Forget about hash collisions because using full hash
   Here is my twjson feed: https://doesnm.p.psf.lt/twjson.json
   And twtxt2json converter: https://doesnm.p.psf.lt/twjson.js

@eapl.me@eapl.me Interesting! Two points stood right out to me:

1. Why the hell are e-mail newsletters considered a valid option in the first place? Just offer an Atom feed and be done with it! Especially for a blog of this very type. This doesn’t even involve a third party service. Although, in addition he also links to Feedburner, what the fuck!? No e-mail address or the like is needed and subject to being disclosed.

2. When these spam mailers want to prevent resubscribing, then for fuck’s sake, why don’t they use a hash of the e-mail address (I saw that in yarnd) for that purpose? Storing the e-mail address in clear text after unsubscribing is illegal in my book.

There are 82.108 read statuses, but only 24.421 messages in the cache. In contrast to the cache with the messages, the read statuses are never cleaned up when a feed was unsubscribed from. And the read statuses also contain old style hashes, before we settled on the what we have today. Still a huge difference. Hmm.

Thanks, @movq@www.uninformativ.de!

My backing SQLite database with indices is 8.7 MiB in size right now.

The twtxt cache is 7.6 MiB, it uses Python’s pickle module. And next to it there is a 16.0 MiB second database with all the read statuses for the old tt. Wow, super inefficient, it shouldn’t contain anything else, it’s a giant, pickled {"$hash": {"read": True/False}, …}. What the heck, why is it so big?! O_o

(Back in tt.) Well, it kinda worked. At least appending to the file. But my cache database got screwed up. I do not yet support replies, so the subject and and root hash columns have not been set at all, resulting in a message that is just not shown at all. I gotta do something about that next. The good thing is, though, after simply fixing the two columns the message appeared on screen.

@bender@twtxt.net Yeah, as you mentioned in the other thread, @andros@twtxt.andros.dev’s hashes appear to be not quite right. 🤔

@andros@twtxt.andros.dev your client is breaking things, I am afraid. This hash (ptxsca), which you seem to be using to reply to @movq@www.uninformativ.de is not the right one.

@movq@www.uninformativ.de I have no doubt that you’re not seeing the images correctly 😀. It’s just that it’s broken when viewing them, in my case, and analyzing the URLs, I’ve seen everything I mentioned.
Regarding the hash, you’re right. I’ll have to investigate what’s going on. I’m having a hard time getting the hash generation to work properly.

@andros@twtxt.andros.dev Hm, looks correct to me. The image to be displayed is a thumbnail and this links to the full-sized image. The thumbnail (JPG) is auto-generated from the full image (PNG), hence the two extensions.

What does look strange, though, is that your client came up with the hash pqsmcka, while it should have been te5quba. 🤔

Why not just use registry? It can be personal or hosted by someone like registry.twtxt.org. Just need to be adapt to support hashes

@prologic@twtxt.net We can’t agree on this idea because that makes things even more complicated than it already is today. The beauty of twtxt is, you put one file on your server, done. One. Not five million. Granted, there might be archive feeds, so it might be already a bit more, but still faaaaaaar less than one file per message.

Also, you would need to host not your own hash files, but everybody else’s as well you follow. Otherwise, what is that supposed to achieve? If people are already following my feed, they know what hashes I have, so this is to no use of them (unless they want to look up a message from an archive feed and don’t process them). But the far more common scenario is that an unknown hash originates from a feed that they have not subscribed to.

Additionally, yarnd’s URL schema would then also break, because https://twtxt.net/twt/<hash> now becomes https://twtxt.net/user/prologic/<hash>, https://twtxt.net/user/bender/<hash> and so on. To me, that looks like you would only get hashes if they belonged to this particular user. Of course, you could define rules that if there is a /user/ part in the path, then use a different URL, but this complicates things even more.

Sorry, I don’t like that idea.

One of the biggest gripes of the community with the way the threading model currently works with Twtxt v1.2 (https://twtxt.dev) is this notion of:

What is this hash?
What does it refer to?

Idea: Why can’t we all agree to implement a simple URI scheme where we host our Twtxt feeds?

That is, if you host your feed at https://example.com/twtxt.txt – Why can’t or could you not also host various JSON files (let’s agree on the spec of course) at https://example.com/twt/<hash> ? 🤔

That way we solve this problem in a truly decentralised way, rather than every relying on yarnd pods alone.

a few async ideas for later

The editing process needs a lot of consideration and compromises.

From one side, editing and deleting it’s necessary IMO. People will do it anyway, and personally I like to edit my texts, so I’d put some effort on make it work.
Should we keep a history of edits? Should we hash every edit to avoid abuse? Should we mark internally a twt as deleted, but keeping the replies?

I think that’s part of a more complete ‘thread’ extension, although I’d say it’s worth to agree on something reflecting the real usage in the wild, along with what people usually do on other platforms.

looks good to me!

About alice’s hash, using SHA256, I get 96473b4f or 96473B4F for the last 8 characters. I’ll add it as an implementation example.
The idea of including it besides the follow URL is to avoid calculating it every time we load the file (assuming the client did that correctly), and helps to track replies across the file with a simple search.

Also, watching your example I’m thinking now that instead of {url=96473B4F,id=1} which is ambiguous of which URL we are referring to, it could be something like:
{reply_to=[URL_HASH]_[TWT_ID]} / {reply_to=96473B4F_1}
That way, the ‘full twt ID’ could be 96473B4F_1.

True. Though if the idea turns out to be better.. then community will adopt it.

if you look at the subject for that twt you will see that it uses the extended hash format to include a URL address.

@bmallred@staystrong.run I forgot one more effect of edits. If clients remember the read status of massages by hash, an edit will mark the updated message as unread again. To some degree that is even the right behavior, because the message was updated, so the user might want to have a look at the updated version. On the other hand, if it’s just a small typo fix, it’s maybe not worth to tell the user about. But the client doesn’t know, at least not with additional logic.

Having said that, it appears that this only affects me personally, noone else. I don’t know of any other client that saves read statuses. But don’t worry about me, all good. Just keep doing what you’ve done so far. I wanted to mention that only for the sake of completeness. :-)

I like this syntax, you have my vote, although I’d change it a bit like
#<Alice https://example.com/twtxt.com#2024-12-18T14:18:26+01:00>

Hashes are not a problem on PHP, I dont know why it’s slow to calculate them from your side, but I agree with your points.

BTW, did you have the chance to read my proposal on twtxt 2.0? I shared a few ideas about possible improvements to discuss:
https://text.eapl.mx/a-few-ideas-for-a-next-twtxt-version
https://text.eapl.mx/reply-to-lyse-about-twtxt

@bmallred@staystrong.run Any edit automatically changes the twt hash, because the hash is built over the hash URL, message timestamp and message text. https://twtxt.dev/exts/twt-hash.html So, it is only a problem, if somebody replied to your original message with the old hash. The original message suddenly doesn’t exist anymore and the reply becomes detached, orphaned, whatever you wanna call it. Threading doesn’t break, though, if nobody replied to your message.

@andros@twtxt.andros.dev I believe you have just reproduced the bug… it looks like you’ve replayed to a twt but the hash is wrong. I can see the hash here from Jenny, but it doesn’t look like it corresponds to any{twt,thing}. if you check it out on any yarn instance it won’t look like a replay.

My hypothesis about that thing breaking my twts is that it might have something to do with the parenthesis surrounding the root twt hash in the replay twt-A when I replay to it with fork-twt-B; I imagine elisp interpreting those as a s-expression thus breaking the generation precess of hash (#twt-A) before prepending it to for-twt-B … but then I’m too ignorant to figure out how to test my theory (heck I couldn’t even recalculate the hashes myself correctly in bash xD). I’ll keep trying tho.