On the Subject of Feed Identities; I propose the following:

1. Generate a Private/Public ED25519 key pair
   
2. Use this key pair to sign your Twtxt feed
   
3. Use it as your feed’s identity in place of # url = as # key = ...
   

For example:

$ ssh-keygen -f prologic@twtxt.net
$ ssh-keygen -Y sign -n prologic@twtxt.net -f prologic@twtxt.net twtxt.txt

And your feed would looke like:

# nick        = prologic
# key         = SHA256:23OiSfuPC4zT0lVh1Y+XKh+KjP59brhZfxFHIYZkbZs
# sig         = twtxt.txt.sig
# prev        = j6bmlgq twtxt.txt/1
# avatar      = https://twtxt.net/user/prologic/avatar#gdoicerjkh3nynyxnxawwwkearr4qllkoevtwb3req4hojx5z43q
# description = "Problems are Solved by Method" 🇦🇺👨‍💻👨‍🦯🏹♔ 🏓⚯ 👨‍👩‍👧‍👧🛥 -- James Mills (operator of twtxt.net / creator of Yarn.social 🧶)

2024-06-14T18:22:17Z	(#nef6byq) @<bender https://twtxt.net/user/bender/twtxt.txt>  Hehe thanks! 😅 Still gotta sort out some other bugs, but that's tomorrows job 🤞
...

Twt Hash extension would change of course to use a feed’s ED25519 public key fingerprint.

@bender@twtxt.net Yes, they do 🤣 Implicitly, or threading would never work at all 😅 Nor lookups 🤣 They are used as keys. Think of them like a primary key in a database or index. I totally get where you’re coming from, but there are trade-offs with using Message/Thread Ids as opposed to Content Addressing (like we do) and I believe we would just encounter other problems by doing so.

My money is on extending the Twt Subject extension to support more (optional) advanced “subjects”; i.e: indicating you edited a Twt you already published in your feed as @falsifian@www.falsifian.org indicated 👌

Then we have a secondary (bure much rarer) problem of the “identity” of a feed in the first place. Using the URL you fetch the feed from as @lyse@lyse.isobeef.org ’s client tt seems to do or using the # url = metadata field as every other client does (according to the spec) is problematic when you decide to change where you host your feed. In fact the spec says:

Users are advised to not change the first one of their urls. If they move their feed to a new URL, they should add this new URL as a new url field.

See Choosing the Feed URL – This is one of our longest debates and challenges, and I think (_I suspect along with @xuu@txt.sour.is _) that the right way to solve this is to use public/private key(s) where you actually have a public key fingerprint as your feed’s unique identity that never changes.

@movq@www.uninformativ.de @prologic@twtxt.net Another option would be: when you edit a twt, prefix the new one with (#[old hash]) and some indication that it’s an edited version of the original tweet with that hash. E.g. if the hash used to be abcd123, the new version should start “(#abcd123) (redit)”.

What I like about this is that clients that don’t know this convention will still stick it in the same thread. And I feel it’s in the spirit of the old pre-hash (subject) convention, though that’s before my time.

I guess it may not work when the edited twt itself is a reply, and there are replies to it. Maybe that could be solved by letting twts have more than one (subject) prefix.

But the great thing about the current system is that nobody can spoof message IDs.

I don’t think twtxt hashes are long enough to prevent spoofing.

All this hash breakage made me wonder if we should try to introduce “message IDs” after all. 😅

But the great thing about the current system is that nobody can spoof message IDs. 🤔 When you think about it, message IDs in e-mails only work because (almost) everybody plays fair. Nothing stops me from using the same Message-ID header in each and every mail, that would break e-mail threading all the time.

In Yarn, twt hashes are derived from twt content and feed metadata. That is pretty elegant and I’d hate see us lose that property.

If we wanted to allow editing twts, we could do something like this:

2024-09-05T13:37:40+00:00   (~mp6ox4a) Hello world!

Here, mp6ox4a would be a “partial hash”: To get the actual hash of this twt, you’d concatenate the feed’s URL and mp6ox4a and get, say, hlnw5ha. (Pretty similar to the current system.) When people reply to this twt, they would have to do this:

2024-09-05T14:57:14+00:00	(~bpt74ka) (<a href="https://yarn.girlonthemoon.xyz/search?q=%23hlnw5ha">#hlnw5ha</a>) Yes, hello!

That second twt has a partial hash of bpt74ka and is a reply to the full hash hlnw5ha. The author of the “Hello world!” twt could then edit their twt and change it to 2024-09-05T13:37:40+00:00 (~mp6ox4a) Hello friends! or whatever. Threading wouldn’t break.

Would this be worth it? It’s certainly not backwards-compatible. 😂

@prologic@twtxt.net Perfect, thanks. For my own future reference: curl -H ‘Accept: application/json’ https://twtxt.net/twt/st3wsda

@prologic@twtxt.net Specifically, I could view yarnd’s copy here, but only as rendered for a human to view: https://twtxt.net/twt/st3wsda

@movq@www.uninformativ.de thanks for getting to the bottom of it. @prologic@twtxt.net is there a way to view yarnd’s copy of the raw twt? The edit didn’t result in a visible change; being able to see what yarnd originally downloaded would have helped me debug.

@bender@twtxt.net On twtxt, I follow all feeds that I can find (there are some exceptions, of course). There’s so little going on in general, it hardly matters. 😅

And I just realized: Mutt’s layout helps a lot. Skimming over new twts is really easy and it’s not a big loss if there are a couple of shitposts™ in my “timeline”. This is very different from Mastodon (both the default web UI and all clients I’ve tried), where the timeline is always huge. Posts take up a lot of space on screen. Makes me think twice if I want to follow someone or not. 😅

(I mostly only follow Hashtags on Mastodon anyway. It’s more interesting that way.)

@prologic@twtxt.net One of your twts begins with (#st3wsda): https://twtxt.net/twt/bot5z4q

Based on the twtxt.net web UI, it seems to be in reply to a twt by @cuaxolotl@sunshinegardens.org which begins “I’ve been sketching out…”.

But jenny thinks the hash of that twt is 6mdqxrq. At least, there’s a very twt in their feed with that hash that has the same text as appears on yarn.social (except with ‘ instead of ’).

Based on this, it appears jenny and yarnd disagree about the hash of the twt, or perhaps the twt was edited (though I can’t see any difference, assuming ’ vs ’ is just a rendering choice).

@prologic@twtxt.net I believe you when you say registries as designed today do not crawl. But when I first read the spec, it conjured in my mind a search engine. Now I don’t know how things work out in practice, but just based on reading, I don’t see why it can’t be an API for a crawling search engine. (In fact I don’t see anything in the spec indicating registry servers shouldn’t crawl.)

(I also noticed that https://twtxt.readthedocs.io/en/latest/user/registry.html recommends “The registries should sync each others user list by using the users endpoint”. If I understood that right, registering with one should be enough to appear on others, even if they don’t crawl.)

Does yarnd provide an API for finding twts? Is it similar?

I just manually followed the steps at https://dev.twtxt.net/doc/twthashextension.html and got 6mdqxrq. I wonder what happened. Did @cuaxolo@sunshinegardens.org edit the twt in some subtle way after twtxt.net downloaded it? I couldn’t spot a diff, other than ‘ appearing as ’ on yarn.social, which I assume is a transformation done by twtxt.net.

@prologic@twtxt.net How does yarn.social’s API fix the problem of centralization? I still need to know whose API to use.

Say I see a twt beginning (#hash) and I want to look up the start of the thread. Is the idea that if that twt is hosted by a a yarn.social pod, it is likely to know the thread start, so I should query that particular pod for the hash? But what if no yarn.social pods are involved?

The community seems small enough that a registry server should be able to keep up, and I can have a couple of others as backups. Or I could crawl the list of feeds followed by whoever emitted the twt that prompted my query.

I have successfully used registry servers a little bit, e.g. to find a feed that mentioned a tag I was interested in. Was even thinking of making my own, if I get bored of my too many other projects :-)

@movq@www.uninformativ.de Thanks, it works!

But when I tried it out on a twt from @prologic@twtxt.net, I discovered jenny and yarn.social seem to disagree about the hash of this twt: https://twtxt.net/twt/st3wsda . jenny assigned it a hash of 6mdqxrq but the URL and prologic’s reply suggest yarn.social thinks the hash is st3wsda. (And as a result, jenny –fetch-context didn’t work on prologic’s twt.)

@movq@www.uninformativ.de I think you are worrying about a non-issue. I see nothing to do on your example twt, because there is no context. Furthermore, if I wanted to follow the feed, everything I need is already on that twt example. :-)

@falsifian@www.falsifian.org @bender@twtxt.net I pushed an alternative implementation to the fetch-context branch. This integrates the whole thing into mutt/jenny.

You will want to configure a new mutt hotkey, similar to the “reply” hotkey:

macro index,pager <esc>C "\
<enter-command> set my_pipe_decode=\$pipe_decode nopipe_decode<Enter>\
<pipe-message> jenny -c<Enter>\
<enter-command> set pipe_decode=\$my_pipe_decode; unset my_pipe_decode<Enter>" \
"Try to fetch context of current twt, like a missing root twt"

This pipes the mail to jenny -c. jenny will try to find the thread hash and the URL and then fetch it. (If there’s no URL or if the specific twt cannot be found in that particular feed, it could query a Yarn pod. That is not yet implemented, though.)

The whole thing looks like this:

https://movq.de/v/0d0e76a180/jenny.mp4

In other words, when there’s a missing root twt, you press a hotkey to fetch it, done.

I think I like this version better. 🤔

(This needs a lot of testing. 😆)

@prologic@twtxt.net Yes, fetching the twt by hash from some service could be a good alternative, in case the twt I have does not @-mention the source. (Besides yarnd, maybe this should be part of the registry API? I don’t see fetch-by-hash in the registry API docs.)

@movq@www.uninformativ.de I don’t know if I’d want to discard the twts. I think what I’m looking for is a command “jenny -g https://host.org/twtxt.txt” to fetch just that one feed, even if it’s not in my follow list. I could wrap that in a shell script so that when I see a twt in reply to a feed I don’t follow, I can just tap a key and the feed will get added to my maildir. I guess the script would look for a mention at the start of a selected twt and call jenny -g on the feed.

@movq@www.uninformativ.de Is there a good way to get jenny to do a one-off fetch of a feed, for when you want to fill in missing parts of a thread? I just added @slashdot@feeds.twtxt.net to my private follow file just because @prologic@twtxt.net keeps responding to the feed :-P and I want to know what he’s commenting on even though I don’t want to see every new slashdot twt.

@falsifian@www.falsifian.org I do like to think of that recommendation as sarcastic stab at twt-errr’s rules 😉 so, IMO the more a twt exceeds the recommended limit the merrier! 😂

Hmmm I’m a little concerned, as I’m seeing quite a few feeds I follow in an error state:

I’m not so concerned with the 15x context deadline exceeded but more concerned with:

aelaraji@aelaraji.com  Unfollow (6 twts, Last fetched 5m ago with error:
dead feed: 403 Forbidden
x4 times.)

And:

anth@a.9srv.net  Unfollow (1 twts, Last fetched 5m ago with error:
Get "http://a.9srv.net/tw.txt": dial tcp 144.202.19.161:80: connect: connection refused
x3733 times.)

Hmmm, maybe the stats are a bit off? 🤔

twts are taking a very long time to post from yarn after the latest upgrade. Like a good 60 seconds.

@prologic@twtxt.net I don’t know if this is new, but I’m seeing:

Jul 25 16:01:17 buc yarnd[1921547]: time="2024-07-25T16:01:17Z" level=error msg="https://yarn.stigatle.no/user/stigatle/twtxt.txt: client.Do fail: Get \"https://yarn.stigatle.no/user/stigatle/twtxt.txt\": dial tcp 185.97.32.18:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" error="Get \"https://yarn.stigatle.no/user/stigatle/twtxt.txt\": dial tcp 185.97.32.18:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"

I no longer see twts from @stigatle@yarn.stigatle.no at all.

This is a test. I am not seeing twts from @stigatle@yarn.stigatle.no and it seems like @prologic@twtxt.net might not be seeing twts from me. Do people see this?

@prologic@twtxt.net I am not seeing twts from @stigatle@yarn.stigatle.no anymore. Are you seeing twts from me?

@prologic@twtxt.net There are a lot of logs being generated by yarnd, which is something I haven’t seen before too:

Jul 25 14:32:42 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:42 (162.211.155.2) "GET /twt/ubhq33a HTTP/1.1" 404 29 643.251µs
Jul 25 14:32:43 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:43 (162.211.155.2) "GET /twt/112073211746755451 HTTP/1.1" 400 12 505.333µs
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (111.119.213.103) "GET /twt/whau6pa HTTP/1.1" 200 37360 35.173255ms
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (162.211.155.2) "GET /twt/112343305123858004 HTTP/1.1" 400 12 455.069µs
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (168.199.225.19) "GET /external?nick=lovetocode999&uri=http%3A%2F%2Fwww.palapa.pl%2Fbaners.php%3Flink%3Dhttps%3A%2F%2Fwww.dwnewstoday.com HTTP/1.1" 200 36167 19.582077ms
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (162.211.155.2) "GET /twt/112503061785024494 HTTP/1.1" 400 12 619.152µs
Jul 25 14:32:46 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:46 (162.211.155.2) "GET /twt/111863876118553837 HTTP/1.1" 400 12 817.678µs
Jul 25 14:32:46 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:46 (162.211.155.2) "GET /twt/112749994821704400 HTTP/1.1" 400 12 540.616µs
Jul 25 14:32:47 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:47 (103.204.109.150) "GET /external?nick=lovetocode999&uri=http%3A%2F%2Fampurify.com%2Fbbs%2Fboard.php%3Fbo_table%3Dfree%26wr_id%3D113858 HTTP/1.1" 200 36187 15.95329ms

I’ve seen that nick=lovetocode999 a bunch.

There are also a bunch of log messages scrolling by. I’ve never seen this much activity in the log:

Jul 25 01:37:39 buc.ci yarnd[829]: [yarnd] 2024/07/25 01:37:39 (149.71.56.69) "GET /external?nick=lovetocode999&uri=https://pagez.co.uk/services/your-own-100-fully-owned-online-vi>
Jul 25 01:37:39 buc.ci yarnd[829]: [yarnd] 2024/07/25 01:37:39 (162.211.155.2) "GET /twt/112135496802692324 HTTP/1.1" 400 12 826.65µs
Jul 25 01:37:40 buc.ci yarnd[829]: [yarnd] 2024/07/25 01:37:40 (51.222.253.14) "GET /conv/muttriq HTTP/1.1" 200 36881 20.448309ms
Jul 25 01:37:40 buc.ci yarnd[829]: [yarnd] 2024/07/25 01:37:40 (162.211.155.2) "GET /twt/112730114943543514 HTTP/1.1" 400 12 663.493µs
Jul 25 01:37:40 buc.ci yarnd[829]: [yarnd] 2024/07/25 01:37:40 (27.75.213.253) "GET /external?nick=lovetocode999&uri=http%3A%2F%2Falfarah.jo%2FHome%2FChangeCulture%3FlangCode%3Den>
Jul 25 01:37:40 buc.ci yarnd[829]: time="2024-07-25T01:37:40Z" level=error msg="http://bynet.com.br/log_envio.asp?cod=335&email=%21%2AEMAIL%2A%21&url=https%3A%2F%2Fwww.almanacar.c>
Jul 25 01:37:40 buc.ci yarnd[829]: [yarnd] 2024/07/25 01:37:40 (162.211.155.2) "GET /twt/111674756400660911 HTTP/1.1" 400 12 545.106µs
Jul 25 01:37:40 buc.ci yarnd[829]: time="2024-07-25T01:37:40Z" level=warning msg="feed FetchFeedRequest: @<lovetocode999 http://alfarah.jo/Home/ChangeCulture?langCode=en&returnUrl>
Jul 25 01:37:41 buc.ci yarnd[829]: [yarnd] 2024/07/25 01:37:41 (162.211.155.2) "GET /twt/112507964696096567 HTTP/1.1" 400 12 838.946µs

Something really weird is going on?

The delete twt is not working.

So updated. Seems to duplicate here in the ui. And what is this “Read More” on every twt now?

By the way, @xuu@txt.sour.is, it looks like you’re running an old, buggy version of yarnd, that duplicates twts in the feed on edit.

Testing something.. can someone mention me in a twt?

@prologic@twtxt.net LOL fare enough! I’ll keep that in mind fir future twts … I hope invidious instances are ok 🤣

@prologic@twtxt.net I was replaying to your twt and thing got too long and messy, so I emailed you. I hope you don’t mind.

This is a test twt to see if :set formatoptions-=t in vim would stop the annoying line breaking I’ve been having in my twts… And I guess, that’s it! Things are looking OK on my end.

How about if I add in a separate paragraph like this one? Did hit return twice for it. I hope it isn’t breaking anything else.

@bender@twtxt.net /me taking notes for future twts …
Sorry for the confusion lads! o.o’

@bender@twtxt.net TBH It was quite late (where I am) when I wrote that twt, way past 2 AM … 😅
But there wasn’t much activity during the day either. It got me wondering if there was something wrong with my cron task for fetching your feeds.

Maybe it’s just the weekends. + there isn’t much bloat content around this space as on the usual platforms anyway.

Is Yarn.social dead or just too niche? (uyrrria) 🧐

Hah 🤣 @dfaria@twtxt.net Your @dfaria.eu@dfaria.eu feed really does consume about >50% of a “Discover” search with filters “Without replies” and “Hide my posts”. 🤣 36/2 = 18 at 25 Twts per page, that’s about ~72% of the search/view real estate you’re taking up! wow 🤩 – I’d be very interested to hear what ideas you have to improve this? Those search filters were created so you could sift through either your own Timeline or the Discover view easily.

@bender@twtxt.net Yeah I know! 😅 I’ve checked their blog and mastodon earlier. That’s what inspired me to replay to their twt, just in case.

I might have found the actual source of my problem.

Jenny uses an .eml file when composing a twt …
and vim kinda auto formats it and inserts in those line breaks every ~70 character.

Then, I stumbled upon this link where Where someone reports that saving a .eml into a .txt might… corrupt the data?

@prologic@twtxt.net Thank
you! and here’s a twt with the said random characters, since I’ve been
cleaning them up manually, earlier before scp-ing my twtxt.txt file. And
maybe a screenshot of how things look in my editor?

Those new lines are added automatically as I type (except for the ones
after the screenshot.

@bender@twtxt.net I’m using both machines in English.

Checked my locale and it spits out:

LANG=en_US.UTF-8
LC_CTYPE=“en_US.UTF-8”
LC_NUMERIC=“en_US.UTF-8”
LC_TIME=“en_US.UTF-8”
LC_COLLATE=“en_US.UTF-8”
LC_MONETARY=“en_US.UTF-8”
LC_MESSAGES=“en_US.UTF-8”
LC_PAPER=“en_US.UTF-8”
LC_NAME=“en_US.UTF-8”
LC_ADDRESS=“en_US.UTF-8”
LC_TELEPHONE=“en_US.UTF-8”
LC_MEASUREMENT=“en_US.UTF-8”
LC_IDENTIFICATION=“en_US.UTF-8”
LC_ALL=

🤷🏽 … and that only happens when vi, vim or nvim are launched by Jenny to compose a twt.

@prologic@twtxt.net it actually does!

But I broke something trying to get rid of the random characters showing on my twts as mentioned here #k7tcqwq.
I taught it was as easy as swapping the ‘\u2028’ in jenny’s new_twt_from_file function but there’s a reason I’m not a developer (yet) 😅

It kinda got rid of them in a way but broke the new lines in the process. So I put things back the way the were till I figure out something else.

@lyse@lyse.isobeef.org Thank you! and sorry, I’ve just noticed your twt. About the funky characters, it’s probably something off with my editor, I’ve just ssh-ed from mobile and checked my .txt file, it looked like that when I cat the file but normal on neoutt. I’ll try and see what’s the deal first thing in the morning. (/me wondering if the same thing would happen with this twt)

Hey @sorenpeter@darch.dk, I’m sorry to tell you, but the prev field in your feed’s headers is invalid. 😅

First, it doesn’t include the hash of the last twt in the archive. Second, and that’s probably more important, it forms an infinite loop: The prev field of your main feed specifies http://darch.dk/twtxt-archive.txt and that file then again specifies http://darch.dk/twtxt-archive.txt. Some clients might choke on this, mine for example. 😂 I’ll push a fix soon, though.

For reference, the prev field is described here: https://dev.twtxt.net/doc/archivefeedsextension.html

It not that easy @xuu@txt.sour.is since I implemented webmentions in a different way that how it have been done in yarnd to work with txt-files. You can find the code in webmention_endpoint.php and new_twt.php at main · sorenpeter/timeline

@shreyan@twtxt.net What do you mean when you say federation protocol?

• Either use webfinger for identity like mastodon etc. or use ATproto from Bluesky (or both?)

• We can use webmentions or create our own twt-mentions for notifying someones feed (WIP code at: https://github.com/sorenpeter/timeline/tree/webmention/views)

I’m not sure we need much else. I would not even bother with encryption since other platforms does that better, and for me twtxt/yarn/timeline is for making things public

Twtxt spec enhancement proposal thread 🧵

Adding attributes to individual twts similar to adding feed attributes in the heading comments.

https://git.mills.io/yarnsocial/go-lextwt/pulls/17

The basic use case would be for multilingual feeds where there is a default language and some twts will be written a different language.

As seen in the wild: https://eapl.mx/twtxt.txt

The attributes are formatted as [key=value]

They can show up in the twt anywhere it is not enclosed by another element such as codeblock or part of a markdown link.

@sorenpeter@darch.dk this makes sense as a quote twt that references a direct URL. If we go back to how it developed on twitter originally it was RT @nick: original text because it contained the original text the twitter algorithm would boost that text into trending.

i like the format (#hash) @<nick url> > "Quoted text"\nThen a comment
as it preserves the human read able. and has the hash for linking to the yarn. The comment part could be optional for just boosting the twt.

The only issue i think i would have would be that that yarn could then become a mess of repeated quotes. Unless the client knows to interpret them as multiple users have reposted/boosted the thread.

The format is also how iphone does reactions to SMS messages with +number liked: original SMS

@eapl.me@eapl.me this is interesting. Is the square bracket something used in the wild for multilingual twts?

@prologic@twtxt.net what are your thoughts? Should we extend the parser to handle [lang] and [boost] ? Or a generic attribute spec. Single word is a boolean attribute. And one with an = is a string key/value.

Un twt pour tester mon nouveau client qui doit m’afficher le nombre de twt publiés