@aelaraji@aelaraji.com Can you give me examples of hashes that you have detected wrong between Emacs client and twtxt.net?
Perhaps there is some character, some space, that is creating the discrepancy.

Why you got to lie to me Google?

@andros@twtxt.andros.dev hmmm… pretty strange, isn’t it? replaying to threads worked perfectly, I’ve only had that problem trying to replay to a twt that was part of a thread.

As an example, this one is a Fork-Replay from Jenny. My next twt will be a replay to this exact twt but from twtxt-el as a test.
Then I’will file an issue if it doesn’t behave the way it’s supposed to. Cheers!

@prologic@twtxt.net Have you tried Google’s robots.txt report? https://support.google.com/webmasters/answer/6062598?hl=en . I would expect Google to be pretty good about this sort of thing. If you have the energy to dig into it and, for example, post on support.google.com, I’d be curious to hear what you find out.

well, Gemini clients like Lagrange allow to show inline images when you click on an image link. Text based clients, like Amfora, usually allow to watch the image in another ‘window’.

For example here: gemini://text.eapl.mx/en-making-a-tic-tac-toe-variant and there https://text.eapl.mx/en-making-a-tic-tac-toe-variant

I agree that some topics require images to make it easier to explain.

@prologic@twtxt.net Of course you don’t notice it when yarnd only shows at most the last n messages of a feed. As an example, check out mckinley’s message from 2023-01-09T22:42:37Z. It has “[Scheduled][Scheduled][Scheduled]“… in it. This text in square brackets is repeated numerous times. If you search his feed for closing square bracket followed by an opening square bracket (][) you will find a bunch more of these. It goes without question he never typed that in his feed. My client saves each twt hash I’ve explicitly marked read. A few days ago, I got plenty of apparently years old, yet suddenly unread messages. Each and every single one of them containing this repeated bracketed text thing. The only conclusion is that something messed up the feed again.

@prologic@twtxt.net In the EU there are Laws, Rules and Regulations for many things. I’m not an expert, but your case may sound like it could match to the EU Digital Services Act.

[…] for example, the obligation to establish points of contact for authorities and citizens […]

Android 16’s Linux Terminal will soon let you run graphical apps, so of course we ran Doom
Regardless, the fact that Android’s Linux Terminal can run graphical apps like Doom now is good news. Hopefully we’ll be able to run more complex desktop-class Linux programs in the future. I tried running GIMP, for example, but it didn’t work. Eventually, Android should be able to run Linux apps as well as Chromebooks can, as I believe one of the goals … ⌘ Read more

a year ago I had a struggle to find documentation about it and now it seems there are more examples, cool!

I updated the specification with base64, Curve25519 and more examples: https://github.com/tanrax/twtxt-direct-message-extension

@prologic@twtxt.net @lyse@lyse.isobeef.org First, please leave me your comments on the repository! Even if it’s just to give your opinion on what shouldn’t be included. The more variety, the better.

Second, I’m going to try to do tests with Elliptic keys and base64. Thanks for the advice @eapl@eapl.me

Finally, I’d like to give my opinion. Secure direct messages are a feature that ActivityPub and Mastodon don’t have, to give an example. By including it as an extension, we’re already taking a significant leap forward from the competition. Does it make sense to include it in a public feed? In fact, we’re already doing that. When we reply to a user, mentioning them at the beginning of the message, it’s already a direct message. The message is within a thread, perhaps breaking the conversation. Direct messages would help isolate conversations between 2 users, as well as keeping a thread cleaner and maintaining privacy. I insist, it’s optional, it doesn’t break compatibility with any client and implementing it isn’t complex. If you don’t like it, you’re free to not use it. If you don’t have a public key, no one can send you direct messages.

@doesnm@doesnm.p.psf.lt I see problems with that, that do not exist on my approach. You could see, example.com/x/bananas/yo.txt, and the feed has no nick. What is the nick?

@prologic@twtxt.net oh it’s ok! thank you for the explanation! i think for me when it comes to programming i learn best by doing, so like written examples or talking about it helps less, BUT baseline explanations like what a pointer is does help! i was so confused and i still need to fix the error i’m having but i will figure it out!

@prologic@twtxt.net Is it possible to interact with twtxt.net from outside? For example, an search API

Showing my nephew around linux… and what’s a better example of text editing in terminal than an actual twt? eh? 😆

I think we could discuss on implementation details like URLs and Handles.

@nick@nick (Masto/Yarn style)
vs
@nick.eapl.me and @eapl.me (Bsky style)

I see, for example, that yarn shows my account as @eapl.me@eapl.me which looks ‘weird’ although it’s not wrong since my domain and my nick are the same. Honestly I like more the Bsky approach as in https://bsky.app/profile/eapl.me for @eapl.me, as when you look for https://eapl.me, it’s my home page.

Also, I didn’t get it completely if you are also proposing a URL standard using subdomains, like https://nick.domain.tls. I only want to point out that these are more difficult to handle from shared hostings, so I’d prefer to also allow https://domain.tls/nick/

after thinking and researching about it, yep, I agree that WebFinger is a good idea.

For example reading here: https://bsky.social/about/blog/4-28-2023-domain-handle-tutorial
I wasn’t considering some scenarios, like multiple accounts for a single domain (See ‘How can I set and manage multiple subdomain handles?’ in the link above)

For Example:

@prologic@twtxt.net sure! I don’t know if this is what you need but, let me give it a try.

• I have Timeline installed, which has an endpoint to process #webmentions. Mine for example is https://aelaraji.com/timeline/webmention which you can find by querying https://aelaraji.com/.well-known/webfinger.
  
• If you mention someone from #Timeline itself, it takes care of querying that and sending in the mention for you.
  
• Otherwise (what I personally do) you could just:
  

curl -i -d 'source=https://twtxt.net/user/prologic/twtxt.txt#:~:text=2024-12-09T01:22:37Z' -d 'target=https://aelaraji.com/twtxt.txt' https://aelaraji.com/timeline/webmention

basically what @sorenpeter@darch.dk mentioned in his article Here.

Afterwards, the mentions are stored in their own mentions.txt feed. The one from the example above looks like this on my Timeline :

Feel free to spam my endpoint if you’d like to give things a try. 👍

[P.S: personally, I don’t seem to get the mentions if I add the Text fragment part to my target]

@aelaraji@aelaraji.com Can you walk me through your testing process here and share example links etc? 🙏

yep, my point is that the txt part is redundant for twtxt

Also a .txt file could be in any format, for example those plans from John Carmak are stored now as .txt -> https://github.com/ESWAT/john-carmack-plan-archive/blob/master/by_day/johnc_plan_19960218.txt
Although being named .plan was expected for them to work, if I recall correctly -> http://www.catb.org/jargon/html/P/plan-file.html

yep, my point is that the txt part is redundant for twtxt

Also a .txt file could be in any format, for example those plans from John Carmak are stored now as .txt -> https://github.com/ESWAT/john-carmack-plan-archive/blob/master/by_day/johnc_plan_19960218.txt
Although being named .plan was expected for them to work, if I recall correctly -> http://www.catb.org/jargon/html/P/plan-file.html

… it even shows @sorenpeter@darch.dk’s article from 2020 in search results

@bender@twtxt.net here:

FROM golang:alpine as builder
ARG version
ENV HTWTXT_VERSION=$version

WORKDIR $GOPATH/pkg/

RUN wget -O htwtxt.tar.gz https://github.com/plomlompom/htwtxt/archive/refs/tags/${HTWTXT_VERSION}.tar.gz
RUN tar xf htwtxt.tar.gz && cd htwtxt-${HTWTXT_VERSION} && go mod init htwtxt && go mod tidy && go install htwtxt

FROM alpine
ARG version
ENV HTWTXT_VERSION=$version

RUN mkdir -p /srv/htwtxt
COPY --from=builder /go/bin/htwtxt /usr/bin/
COPY --from=builder /go/pkg/htwtxt-${HTWTXT_VERSION}/templates/* /srv/htwtxt/templates/
WORKDIR /srv/htwtxt
VOLUME /srv/htwtxt

EXPOSE 8000

ENTRYPOINT ["htwtxt", "-dir", "/srv/htwtxt", "-templates", "/srv/htwtxt/templates"]

Don’t forget the --build-arg version="1.0.7" for example when building this one, although there isn’t much difference between the couple last versions.
P.S: I may have effed up changing htwtxt’s files directory to /srv/htwtxt when the command itself defaults to /root/htwtxt so you’ll have to throw in a -dir whenever you issue an htwtxt command (i.e: htwtxt -adduser somename:somepwd -dir /srv/htwtxt … etc)

for example, ejabberd, redka, and litefs. all using sqlite+litefs for their database needs allows agents to communicate over xmpp, matrix, mqtt, and sip. other applications can use sqlite for storage or speak the redis protocol to redka. ejabberd can also handle file uploads, static file publishing, identity, and various other web application services. when scaling, litefs integrates with consul to manage replication which grants the network access to service disco, encrypted mesh networking, and various other features that can be used to build secure service grids. ejabberd and redka can be scaled to multiple nodes that coordinate over the litefs replication protocol without any changes to the db storage config. other components can be configured to plug into this framework fairly easily as well. we keep the network config fairly simple by linking nodes together with yggdrasil to flatten the address space and then linking app nodes together using consul to provide secure routing for the local grid service. yggdrasil also offers utility for buliding federated networks in a similarly flat address space, for more secure communications i2p is also available in yggdrasil mode. minibase is wonderful, and we have not even started to talk about secure IoT.

@doesnm@doesnm.p.psf.lt here! create a $HOME/.cache/mutt/twtxt/ directory for example and then add this set header_cache = $HOME/.cache/mutt/twtxt/ to your muttrc (the one you have set up for or use with jenny if you’re using different ones). That’s what helped me with that.

Ref: http://www.mutt.org/doc/manual/#header-caching

For example, this links to the previous post, no crypto needed: http://a.9srv.net/tw.txt#:~:text=2024-10-23T18:59:49-07:00

The V: pattern itself is quite good because you can do quite a lot of powerful things with selected text.

For example: ggV}:s/^/ -/ will insert a - at the beginning of every line turning your bunch of lines into a Markdown list of items 😅

If we stuck with Blake2b for Twt Hash(es); what do we think we need to reasonably go to in bit length/size?

=> https://gist.mills.io/prologic/194993e7db04498fa0e8d00a528f7be6

e.g: (turns out @xuu@txt.sour.is is right about Blak2b being easy/simple too!):

$ printf "%s\t%s\t%s" "https://example.com/twtxt.txt" "2024-09-29T13:30:00Z" "Hello World!" | b2sum -l 32 -t | awk '{ print $1 }'
7b8b79dd

@prologic@twtxt.net Regarding the new way of generating twt-hashes, to me it makes more sense to use tabs as separator instead of spaces, since the you can just copy/past a line directly from a twtxt-file that already go a tab between timestamp and message. But tabs might be hard to “type” when you are in a terminal, since it will activate autocomplete…🤔

Another thing, it seems that you sugget we only use the domain in the hash-creation and not the full path to the twtxt.txt

$ echo -e "https://example.com 2024-09-29T13:30:00Z Hello World!" | sha256sum - | awk '{ print $1 }' | base64 | head -c 12

Diving into mblaze, I think I’ve nearly* reached peek email geek.

Just a bunch of shell commands I can pipe together to search, list, view and reply to email (after syncing it to a local Maildir).

EXAMPLES at https://git.vuxu.org/mblaze/tree/README

So far I’m using most of the tools directly from the command line, but I might take inspiration from https://sr.ht/~rakoo/omail/ to make my workflow a bit more efficient.

*To get any closer, I think I’d have to hand-craft my own SMTP client or something.

@anth@a.9srv.net you wrote:

“Edits and Deletions should go; see also Section 6. This is probably the worst example of this document pushing a text document to do more protocol-like things.”

Edit and deletions are precisely what brought us here. Currently, if one replies to a twtxt, and the original gets later edited, it breaks replies, and potentially drastically changes context.

There is also a ~5x increase cost in memory utilization for any implementations or implementors that use or wish to use in-memory storage (yarnd does for example) and equally a 5x increase in on-disk storage as well. This is based on the Twt Hash going from a 13 bytes (content-addressing) to 63 bytes (on average for location-based addressing). There is roughly a ~20-150% increase in the size of individual feeds as well that needs to be taken into consideration (on the average case).

#fzf is the new emacs: a tool with a simple purpose that has evolved to include an #email client. https://sr.ht/~rakoo/omail/

I’m being a little silly, of course. fzf doesn’t actually check your email, but it appears to be basically the whole user interface for that mail program, with #mblaze wrangling the emails.

I’ve been thinking about how I handle my email, and am tempted to make something similar. (When I originally saw this linked the author was presenting it as an example tweaked to their own needs, encouraging people to make their own.)

This approach could surely also be combined with #jenny, taking the place of (neo)mutt. For example mblaze’s mthread tool presents a threaded discussion with indentation.

@xuu@txt.sour.is I think it is more tricky than that.

https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/application-regulation/who-does-data-protection-law-apply_en

“A company or entity …”

Also, as I understand it, “personal or household activity” (as you called it) is rather strict: An example could be you uploading photos to a webspace behind HTTP basic auth and sending that link to a friend. So, yes, a webserver is involved and you process your friend’s data (e.g., when did he access your files), but it’s just between you and him. But if you were to publish these photos publicly on a webserver that anyone can access, then it’s a different story – even though you could say that “this is just my personal hobby, not related to any job or money”.

If you operate a public Yarn pod and if you accept registrations from other users, then I’m pretty sure the GDPR applies. 🤔 You process personal data and you don’t really know these people. It’s not a personal/private thing anymore.

@prologic@twtxt.net Thanks for writing that up!

I hope it can remain a living document (or sequence of draft revisions) for a good long time while we figure out how this stuff works in practice.

I am not sure how I feel about all this being done at once, vs. letting conventions arise.

For example, even today I could reply to twt abc1234 with “(#abc1234) Edit: …” and I think all you humans would understand it as an edit to (#abc1234). Maybe eventually it would become a common enough convention that clients would start to support it explicitly.

Similarly we could just start using 11-digit hashes. We should iron out whether it’s sha256 or whatever but there’s no need get all the other stuff right at the same time.

I have similar thoughts about how some users could try out location-based replies in a backward-compatible way (append the replyto: stuff after the legacy (#hash) style).

However I recognize that I’m not the one implementing this stuff, and it’s less work to just have everything determined up front.

Misc comments (I haven’t read the whole thing):

• Did you mean to make hashes hexadecimal? You lose 11 bits that way compared to base32. I’d suggest gaining 11 bits with base64 instead.

• “Clients MUST preserve the original hash” — do you mean they MUST preserve the original twt?

• Thanks for phrasing the bit about deletions so neutrally.

• I don’t like the MUST in “Clients MUST follow the chain of reply-to references…”. If someone writes a client as a 40-line shell script that requires the user to piece together the threading themselves, IMO we shouldn’t declare the client non-conforming just because they didn’t get to all the bells and whistles.

• Similarly I don’t like the MUST for user agents. For one thing, you might want to fetch a feed without revealing your identty. Also, it raises the bar for a minimal implementation (I’m again thinking again of the 40-line shell script).

• For “who follows” lists: why must the long, random tokens be only valid for a limited time? Do you have a scenario in mind where they could leak?

• Why can’t feeds be served over HTTP/1.0? Again, thinking about simple software. I recently tried implementing HTTP/1.1 and it wasn’t too bad, but 1.0 would have been slightly simpler.

• Why get into the nitty-gritty about caching headers? This seems like generic advice for HTTP servers and clients.

• I’m a little sad about other protocols being not recommended.

• I don’t know how I feel about including markdown. I don’t mind too much that yarn users emit twts full of markdown, but I’m more of a plain text kind of person. Also it adds to the length. I wonder if putting a separate document would make more sense; that would also help with the length.

@prologic@twtxt.net Wikipedia claims sha1 is vulnerable to a “chosen-prefix attack”, which I gather means I can write any two twts I like, and then cause them to have the exact same sha1 hash by appending something. I guess a twt ending in random junk might look suspcious, but perhaps the junk could be worked into an image URL like . If that’s not possible now maybe it will be later.

git only uses sha1 because they’re stuck with it: migrating is very hard. There was an effort to move git to sha256 but I don’t know its status. I think there is progress being made with Game Of Trees, a git clone that uses the same on-disk format.

I can’t imagine any benefit to using sha1, except that maybe some very old software might support sha1 but not sha256.

@movq@www.uninformativ.de Agreed that hashes have a benefit. I came up with a similar example where when I twted about an 11-character hash collision. Perhaps hashes could be made optional somehow. Like, you could use the “replyto” idea and then additionally put a hash somewhere if you want to lock in which version of the twt you are replying to.

I have noticed that twtxt timestamps differ. For example:

• @prologic@twtxt.net (and I assume any Yarn user)
  2024-09-18T13:16:17Z
  
• @lyse@lyse.isobeef.org
  2024-09-17T21:15:00+02:00
  
• @aelaraji@aelaraji.com (and @movq@www.uninformativ.de, and me)
  2024-09-18T05:43:13+00:00
  

So, which is right, or best?

An alternate idea for supporting (properly) Twt Edits is to denoate as such and extend the meaning of a Twt Subject (which would need to be called something better?); For example, let’s say I produced the following Twt:

2024-09-18T23:08:00+10:00	Hllo World

And my feed’s URI is https://example.com/twtxt.txt. The hash for this Twt is therefore 229d24612a2:

$ echo -n "https://example.com/twtxt.txt\n2024-09-18T23:08:00+10:00\nHllo World" | sha1sum | head -c 11
229d24612a2

You wish to correct your mistake, so you make an amendment to that Twt like so:

2024-09-18T23:10:43+10:00	(edit:#229d24612a2) Hello World

Which would then have a new Twt hash value of 026d77e03fa:

$ echo -n "https://example.com/twtxt.txt\n2024-09-18T23:10:43+10:00\nHello World" | sha1sum | head -c 11
026d77e03fa

Clients would then take this edit:#229d24612a2 to mean, this Twt is an edit of 229d24612a2 and should be replaced in the client’s cache, or indicated as such to the user that this is the intended content.

This might be quite unpopular, but I truly dislike Wordle. The reason isn’t rooted on any psychological issue, it is much, much more simple: people share their Wordle result(s)—I figure they feel good about themselves—and for me it is only uneven, unaligned, wasteful noise. I don’t even want to show you an example, but I am sure you know what I am talking about.

Thank gods those posting their hideous squares have finally quieted down. LOL.

@bender@twtxt.net that’s not your change, silly robot, it is mine! LOL. I am finding @prologic@twtxt.net’s tool handy to refer to previous posts (as reference, for example).

@quark@ferengi.one here is an example: This Thread is not showing up in Mutt 🤔 Something is off!

I’ll set up jenny and mutt on another computer and see how it goes from there.

@prologic@twtxt.net earlier you suggested extending hashes to 11 characters, but here’s an argument that they should be even longer than that.

Imagine I found this twt one day at https://example.com/twtxt.txt :

2024-09-14T22:00Z Useful backup command: rsync -a “$HOME” /mnt/backup

and I responded with “(#5dgoirqemeq) Thanks for the tip!”. Then I’ve endorsed the twt, but it could latter get changed to

2024-09-14T22:00Z Useful backup command: rm -rf /some_important_directory

which also has an 11-character base32 hash of 5dgoirqemeq. (I’m using the existing hashing method with https://example.com/twtxt.txt as the feed url, but I’m taking 11 characters instead of 7 from the end of the base32 encoding.)

That’s what I meant by “spoofing” in an earlier twt.

I don’t know if preventing this sort of attack should be a goal, but if it is, the number of bits in the hash should be at least two times log2(number of attempts we want to defend against), where the “two times” is because of the birthday paradox.

Side note: current hashes always end with “a” or “q”, which is a bit wasteful. Maybe we should take the first N characters of the base32 encoding instead of the last N.

Code I used for the above example: https://fossil.falsifian.org/misc/file?name=src/twt_collision/find_collision.c
I only needed to compute 43394987 hashes to find it.

@mckinley@twtxt.net

HTTPS is supposed to do [verification] anyway.

TLS provides verification that nobody is tampering with or snooping on your connection to a server. It doesn’t, for example, verify that a file downloaded from server A is from the same entity as the one from server B.

I was confused by this response for a while, but now I think I understand what you’re getting at. You are pointing out that with signed feeds, I can verify the authenticity of a feed without accessing the original server, whereas with HTTPS I can’t verify a feed unless I download it myself from the origin server. Is that right?

I.e. if the HTTPS origin server is online and I don’t mind taking the time and bandwidth to contact it, then perhaps signed feeds offer no advantage, but if the origin server might not be online, or I want to download a big archive of lots of feeds at once without contacting each server individually, then I need signed feeds.

feed locations [being] URLs gives some flexibility

It does give flexibility, but perhaps we should have made them URIs instead for even more flexibility. Then, you could use a tag URI, urn:uuid:*, or a regular old URL if you wanted to. The spec seems to indicate that the url tag should be a working URL that clients can use to find a copy of the feed, optionally at multiple locations. I’m not very familiar with IP{F,N}S but if it ensures you own an identifier forever and that identifier points to a current copy of your feed, it could be a great way to fix it on an individual basis without breaking any specs :)

I’m also not very familiar with IPFS or IPNS.

I haven’t been following the other twts about signatures carefully. I just hope whatever you smart people come up with will be backwards-compatible so it still works if I’m too lazy to change how I publish my feed :-)

I’m not sure if it’s possible to have unwind listening on a routing table != 0. It would be handy with my wireguard vpn set up on rdomain 2 (as example) si I can resolve domain names without setting up public DNS server in /etc/resolv.conf #openbsd.

how little data is needed for generating the hashes? Instead of the full URL, can we makedo with just the domain (example.net) so we avoid the conflicts with gemini://, https:// and only http:// (like in my own twtxt.txt) or construct something like like a webfinger id nick@domain (also used by mastodon etc.) from the domain and nick if there, else use domain as nick as well

@lyse@lyse.isobeef.org This looks like a nice way to do it.

Another thought: if clients can’t agree on the url (for example, if we switch to this new way, but some old clients still do it the old way), that could be mitigated by computing many hashes for each twt: one for every url in the feed. So, if a feed has three URLs, every twt is associated with three hashes when it comes time to put threads together.

A client stills need to choose one url to use for the hash when composing a reply, but this might add some breathing room if there’s a period when clients are doing different things.

(From what I understand of jenny, this would be difficult to implement there since each pseudo-email can only have one msgid to match to the in-reply-to headers. I don’t know about other clients.)

@falsifian@www.falsifian.org In my opinion it was a mistake that we defined the first url field in the feed to define the URL for hashing. It should have been the last encountered one. Then, assuming append-style feeds, you could override the old URL with a new one from a certain point on:

# url = https://example.com/alias/txtxt.txt
# url = https://example.com/initial/twtxt.txt
<message 1 uses the initial URL>
<message 2 uses the initial URL, too>
# url = https://example.com/new/twtxt.txt
<message 3 uses the new URL>
# url = https://example.com/brand-new/twtxt.txt
<message 4 uses the brand new URL>

In theory, the same could be done for prepend-style feeds. They do exist, I’ve come around them. The parser would just have to calculate the hashes afterwards and not immediately.

On the Subject of Feed Identities; I propose the following:

1. Generate a Private/Public ED25519 key pair
   
2. Use this key pair to sign your Twtxt feed
   
3. Use it as your feed’s identity in place of # url = as # key = ...
   

For example:

$ ssh-keygen -f prologic@twtxt.net
$ ssh-keygen -Y sign -n prologic@twtxt.net -f prologic@twtxt.net twtxt.txt

And your feed would looke like:

# nick        = prologic
# key         = SHA256:23OiSfuPC4zT0lVh1Y+XKh+KjP59brhZfxFHIYZkbZs
# sig         = twtxt.txt.sig
# prev        = j6bmlgq twtxt.txt/1
# avatar      = https://twtxt.net/user/prologic/avatar#gdoicerjkh3nynyxnxawwwkearr4qllkoevtwb3req4hojx5z43q
# description = "Problems are Solved by Method" 🇦🇺👨‍💻👨‍🦯🏹♔ 🏓⚯ 👨‍👩‍👧‍👧🛥 -- James Mills (operator of twtxt.net / creator of Yarn.social 🧶)

2024-06-14T18:22:17Z	(#nef6byq) @<bender https://twtxt.net/user/bender/twtxt.txt>  Hehe thanks! 😅 Still gotta sort out some other bugs, but that's tomorrows job 🤞
...

Twt Hash extension would change of course to use a feed’s ED25519 public key fingerprint.