@prologic@twtxt.net earlier you suggested extending hashes to 11 characters, but hereās an argument that they should be even longer than that.
Imagine I found this twt one day at https://example.com/twtxt.txt :
2024-09-14T22:00Z Useful backup command: rsync -a ā$HOMEā /mnt/backup
and I responded with ā(#5dgoirqemeq) Thanks for the tip!ā. Then Iāve endorsed the twt, but it could latter get changed to
2024-09-14T22:00Z Useful backup command: rm -rf /some_important_directory
which also has an 11-character base32 hash of 5dgoirqemeq. (Iām using the existing hashing method with https://example.com/twtxt.txt as the feed url, but Iām taking 11 characters instead of 7 from the end of the base32 encoding.)
Thatās what I meant by āspoofingā in an earlier twt.
I donāt know if preventing this sort of attack should be a goal, but if it is, the number of bits in the hash should be at least two times log2(number of attempts we want to defend against), where the ātwo timesā is because of the birthday paradox.
Side note: current hashes always end with āaā or āqā, which is a bit wasteful. Maybe we should take the first N characters of the base32 encoding instead of the last N.
Code I used for the above example: https://fossil.falsifian.org/misc/file?name=src/twt_collision/find_collision.c
I only needed to compute 43394987 hashes to find it.
url
field in the feed to define the URL for hashing. It should have been the last encountered one. Then, assuming append-style feeds, you could override the old URL with a new one from a certain point on:
I was not suggesting to that everyone need to setup a working webfinger endpoint, but that we take the format of nick+(sub)domain as base for generating the hashed together with the message date and content.
If we omit the protocol prefix from the way we do things now will that not solve most of the problems? In the case of gemini://gemini.ctrl-c.club/~nristen/twtxt.txt
they also have a working twtxt.txt at https://ctrl-c.club/~nristen/twtxt.txt
⦠damn I just notice the gemini.
subdomain.
Okay what about defining a prefers protocol as part of the hash schema? so 1: https , 2: http 3: gemini 4: gopher ?
@sorenpeter@darch.dk There was a client that would generate a unique hash for each twt. It didnāt get wide adoption.
So this is a great thread. I have been thinking about this too.. and what if we are coming at it from the wrong direction? Identity being tied to a given URL has always been a pain point. If i get a new URL its almost as if i have a new identity because not only am I serving at a new location but all my previous communications are broken because the hashes are all wrong.
What if instead we used this idea of signatures to thread the URLs together into one identity? We keep the URL to Hash in place. Changing that now is basically a no go. But we can create a signature chain that can link identities together. So if i move to a new URL i update the chain hosted by my primary identity to include the new URL. If i have an archived feed that the old URL is now dead, we can point to where it is now hosted and use the current convention of hashing based on the first url:
The signature chain can also be used to rotate to new keys over time. Just sign in a new key or revoke an old one. The prior signatures remain valid within the scope of time the signatures were made and the keys were active.
The signature file can be hosted anywhere as long as it can be fetched by a reasonable protocol. So say we could use a webfinger that directs to the signature file? you have an identity like frank@beans.co
that will discover a feed at some URL and a signature chain at another URL. Maybe even include the most recent signing key?
From there the client can auto discover old feeds to link them together into one complete timeline. And the signatures can validate that its all correct.
I like the idea of maybe putting the chain in the feed preamble and keeping the single self contained file.. but wonder if that would cause lots of clutter? The signature chain would be something like a log with what is changing (new key, revoke, add url) and a signature of the change + the previous signature.
# chain: ADDKEY kex14zwrx68cfkg28kjdstvcw4pslazwtgyeueqlg6z7y3f85h29crjsgfmu0w
# sig: BEGIN SALTPACK SIGNED MESSAGE. ...
# chain: ADDURL https://txt.sour.is/user/xuu
# sig: BEGIN SALTPACK SIGNED MESSAGE. ...
# chain: REVKEY kex14zwrx68cfkg28kjdstvcw4pslazwtgyeueqlg6z7y3f85h29crjsgfmu0w
# sig: ...
@mckinley@twtxt.net To answer some of your questions:
Are SSH signatures standardized and are there robust software libraries that can handle them? Weāll need a library in at least Python and Go to provide verified feed support with the currently used clients.
We already have this. Ed25519 libraries exist for all major languages. Aside from using ssh-keygen -Y sign
and ssh-keygen -Y verify
, you can also use the salty
CLI itself (https://git.mills.io/prologic/salty), and Iām sure there are other command-line tools that could be used too.
If we all implemented this, every twt hash would suddenly change and every conversation thread weāve ever had would at least lose its opening post.
Yes. This would happen, so weād have to make a decision around this, either a) a cut-off point or b) some way to progressively transition.
url
field in the feed to define the URL for hashing. It should have been the last encountered one. Then, assuming append-style feeds, you could override the old URL with a new one from a certain point on:
how little data is needed for generating the hashes? Instead of the full URL, can we makedo with just the domain (example.net) so we avoid the conflicts with gemini://
, https://
and only http://
(like in my own twtxt.txt) or construct something like like a webfinger id nick@domain
(also used by mastodon etc.) from the domain and nick if there, else use domain as nick as well
@lyse@lyse.isobeef.org This looks like a nice way to do it.
Another thought: if clients canāt agree on the url (for example, if we switch to this new way, but some old clients still do it the old way), that could be mitigated by computing many hashes for each twt: one for every url in the feed. So, if a feed has three URLs, every twt is associated with three hashes when it comes time to put threads together.
A client stills need to choose one url to use for the hash when composing a reply, but this might add some breathing room if thereās a period when clients are doing different things.
(From what I understand of jenny, this would be difficult to implement there since each pseudo-email can only have one msgid to match to the in-reply-to headers. I donāt know about other clients.)
@falsifian@www.falsifian.org In my opinion it was a mistake that we defined the first url
field in the feed to define the URL for hashing. It should have been the last encountered one. Then, assuming append-style feeds, you could override the old URL with a new one from a certain point on:
# url = https://example.com/alias/txtxt.txt
# url = https://example.com/initial/twtxt.txt
<message 1 uses the initial URL>
<message 2 uses the initial URL, too>
# url = https://example.com/new/twtxt.txt
<message 3 uses the new URL>
# url = https://example.com/brand-new/twtxt.txt
<message 4 uses the brand new URL>
In theory, the same could be done for prepend-style feeds. They do exist, Iāve come around them. The parser would just have to calculate the hashes afterwards and not immediately.
@movq@www.uninformativ.de Another idea: just hash the feed url and time, without the message content. And donāt twt more than once per second.
Maybe you could even just use the time, and rely on @-mentions to disambiguate. Not sure how that would work out.
Though I kind of like the idea of twts being immutable. At least, itās clear which version of a twt youāre replying to (assuming nobody is engineering hash collisions).
On the Subject of Feed Identities; I propose the following:
- Generate a Private/Public ED25519 key pair
- Use this key pair to sign your Twtxt feed
- Use it as your feedās identity in place of
# url =
as# key = ...
For example:
$ ssh-keygen -f prologic@twtxt.net
$ ssh-keygen -Y sign -n prologic@twtxt.net -f prologic@twtxt.net twtxt.txt
And your feed would looke like:
# nick = prologic
# key = SHA256:23OiSfuPC4zT0lVh1Y+XKh+KjP59brhZfxFHIYZkbZs
# sig = twtxt.txt.sig
# prev = j6bmlgq twtxt.txt/1
# avatar = https://twtxt.net/user/prologic/avatar#gdoicerjkh3nynyxnxawwwkearr4qllkoevtwb3req4hojx5z43q
# description = "Problems are Solved by Method" š¦šŗšØāš»šØāš¦Æš¹ā šāÆ šØāš©āš§āš§š„ -- James Mills (operator of twtxt.net / creator of Yarn.social š§¶)
2024-06-14T18:22:17Z (#nef6byq) @<bender https://twtxt.net/user/bender/twtxt.txt> Hehe thanks! š
Still gotta sort out some other bugs, but that's tomorrows job š¤
...
Twt Hash extension would change of course to use a feedās ED25519 public key fingerprint.
@bender@twtxt.net Sorry, trust was the wrong word. Trust as in, you do not have to check with anything or anyone that the hash is valid. You can verify the hash is valid by recomputing the hash from the content of what it points to, etc.
@movq@www.uninformativ.de @prologic@twtxt.net Another option would be: when you edit a twt, prefix the new one with (#[old hash]) and some indication that itās an edited version of the original tweet with that hash. E.g. if the hash used to be abcd123, the new version should start ā(#abcd123) (redit)ā.
What I like about this is that clients that donāt know this convention will still stick it in the same thread. And I feel itās in the spirit of the old pre-hash (subject) convention, though thatās before my time.
I guess it may not work when the edited twt itself is a reply, and there are replies to it. Maybe that could be solved by letting twts have more than one (subject) prefix.
But the great thing about the current system is that nobody can spoof message IDs.
I donāt think twtxt hashes are long enough to prevent spoofing.
All this hash breakage made me wonder if we should try to introduce āmessage IDsā after all. š
But the great thing about the current system is that nobody can spoof message IDs. š¤ When you think about it, message IDs in e-mails only work because (almost) everybody plays fair. Nothing stops me from using the same Message-ID
header in each and every mail, that would break e-mail threading all the time.
In Yarn, twt hashes are derived from twt content and feed metadata. That is pretty elegant and Iād hate see us lose that property.
If we wanted to allow editing twts, we could do something like this:
2024-09-05T13:37:40+00:00 (~mp6ox4a) Hello world!
Here, mp6ox4a
would be a āpartial hashā: To get the actual hash of this twt, youād concatenate the feedās URL and mp6ox4a
and get, say, hlnw5ha
. (Pretty similar to the current system.) When people reply to this twt, they would have to do this:
2024-09-05T14:57:14+00:00 (~bpt74ka) (<a href="https://yarn.girlonthemoon.xyz/search?q=%23hlnw5ha">#hlnw5ha</a>) Yes, hello!
That second twt has a partial hash of bpt74ka
and is a reply to the full hash hlnw5ha
. The author of the āHello world!ā twt could then edit their twt and change it to 2024-09-05T13:37:40+00:00 (~mp6ox4a) Hello friends!
or whatever. Threading wouldnāt break.
Would this be worth it? Itās certainly not backwards-compatible. š
@prologic@twtxt.net One of your twts begins with (#st3wsda): https://twtxt.net/twt/bot5z4q
Based on the twtxt.net web UI, it seems to be in reply to a twt by @cuaxolotl@sunshinegardens.org which begins āIāve been sketching outā¦ā.
But jenny thinks the hash of that twt is 6mdqxrq. At least, thereās a very twt in their feed with that hash that has the same text as appears on yarn.social (except with ā instead of ā).
Based on this, it appears jenny and yarnd disagree about the hash of the twt, or perhaps the twt was edited (though I canāt see any difference, assuming ā vs ā is just a rendering choice).
@prologic@twtxt.net How does yarn.socialās API fix the problem of centralization? I still need to know whose API to use.
Say I see a twt beginning (#hash) and I want to look up the start of the thread. Is the idea that if that twt is hosted by a a yarn.social pod, it is likely to know the thread start, so I should query that particular pod for the hash? But what if no yarn.social pods are involved?
The community seems small enough that a registry server should be able to keep up, and I can have a couple of others as backups. Or I could crawl the list of feeds followed by whoever emitted the twt that prompted my query.
I have successfully used registry servers a little bit, e.g. to find a feed that mentioned a tag I was interested in. Was even thinking of making my own, if I get bored of my too many other projects :-)
@movq@www.uninformativ.de Thanks, it works!
But when I tried it out on a twt from @prologic@twtxt.net, I discovered jenny and yarn.social seem to disagree about the hash of this twt: https://twtxt.net/twt/st3wsda . jenny assigned it a hash of 6mdqxrq but the URL and prologicās reply suggest yarn.social thinks the hash is st3wsda. (And as a result, jenny āfetch-context didnāt work on prologicās twt.)
Because I saw the nick on movq
(@prologic@twtxt.net, canāt mention anyone outside this pod, by the way), I looked the user up: https://tilde.pt/~marado/twtxt.txt. I wonder if the āhashesā they are using will work out of the box with jenny
.
Talking about jenny
, going to play with the latest now. Tata! :-)
@falsifian@www.falsifian.org @bender@twtxt.net I pushed an alternative implementation to the fetch-context
branch. This integrates the whole thing into mutt/jenny.
You will want to configure a new mutt hotkey, similar to the āreplyā hotkey:
macro index,pager <esc>C "\
<enter-command> set my_pipe_decode=\$pipe_decode nopipe_decode<Enter>\
<pipe-message> jenny -c<Enter>\
<enter-command> set pipe_decode=\$my_pipe_decode; unset my_pipe_decode<Enter>" \
"Try to fetch context of current twt, like a missing root twt"
This pipes the mail to jenny -c
. jenny will try to find the thread hash and the URL and then fetch it. (If thereās no URL or if the specific twt cannot be found in that particular feed, it could query a Yarn pod. That is not yet implemented, though.)
The whole thing looks like this:
https://movq.de/v/0d0e76a180/jenny.mp4
In other words, when thereās a missing root twt, you press a hotkey to fetch it, done.
I think I like this version better. š¤
(This needs a lot of testing. š)
@prologic@twtxt.net Yes, fetching the twt by hash from some service could be a good alternative, in case the twt I have does not @-mention the source. (Besides yarnd, maybe this should be part of the registry API? I donāt see fetch-by-hash in the registry API docs.)
@movq@www.uninformativ.de you said you liked seeing the hash (which is a fair choice!). All I am asking is for a reconsideration as a user configurable feature. ;-) It looks redundant, in my opinion.
@movq@www.uninformativ.de, that would be a nice addition. :-) I would also love the ability to hide/not show the hash when reading twtxts (after all, thatās on the header on each āemailā). Could that be added as a user configurable toggle?
A equivalent yarnc debug <url>
only sees the 2nd hash
Hey @sorenpeter@darch.dk, Iām sorry to tell you, but the prev
field in your feedās headers is invalid. š
First, it doesnāt include the hash of the last twt in the archive. Second, and thatās probably more important, it forms an infinite loop: The prev
field of your main feed specifies http://darch.dk/twtxt-archive.txt and that file then again specifies http://darch.dk/twtxt-archive.txt. Some clients might choke on this, mine for example. š Iāll push a fix soon, though.
For reference, the prev
field is described here: https://dev.twtxt.net/doc/archivefeedsextension.html
password is generated using caddy hash-password
yarnd does not do auto discovery via webfinger though.. i cant put @username and have it fetch the feed url from webfinger. to fully make feeds portable. would also need to be able to use that for hashing.
>
?
@sorenpeter@darch.dk this makes sense as a quote twt that references a direct URL. If we go back to how it developed on twitter originally it was RT @nick: original text
because it contained the original text the twitter algorithm would boost that text into trending.
i like the format (#hash) @<nick url> > "Quoted text"\nThen a comment
as it preserves the human read able. and has the hash for linking to the yarn. The comment part could be optional for just boosting the twt.
The only issue i think i would have would be that that yarn could then become a mess of repeated quotes. Unless the client knows to interpret them as multiple users have reposted/boosted the thread.
The format is also how iphone does reactions to SMS messages with +number liked: original SMS
@prologic@twtxt.net Iāve even added the twthash message hash to my Twtxt bash CLI script so I can properly answer here.
@prologic@twtxt.net was this in reply to a different thread? Or maybe a hash collision?
MƔs fixes por acƔ en twtxt-php
Ya funciona la vista de Hashes y respuestas!
(Intenta responderme algo) āļø
Started with
a concept sketch of a full body end-time factory worker on a distant planet, cyberpunk light brown suite, (badass), looking up at the viewer, 2d, line drawing, (pencil sketch:0.3), (caricature:0.2), watercolor city sketch,
Negative prompt: EasyNegativ, bad-hands-5, 3d, photo, naked, sexy, disproportionate, ugly
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 2479087078, Face restoration: GFPGAN, Size: 512x768, Model hash: 2ee2a2bf90, Model: mimic_v10, Denoising strength: 0.7, Hires upscale: 1.5, Hires upscaler: Latent
Iām not super a fan of using json. I feel we could still use text as the medium. Maybe a modified version to fix any weakness.
What if instead of signing each twt individually we generated a merkle tree using the twt hashes? Then a signature of the root hash. This would ensure the full stream of twts are intact with a minimal overhead. With the added bonus of helping clients identify missing twts when syncing/gossiping.
Have two endpoints. One as the webfinger to link profile details and avatar like you posted. And the signature for the merkleroot twt. And the other a pageable stream of twts. Or individual twts/merkle branch to incrementally access twt feeds.
š” Quick ān Dirty prototype Yarn.social protocol/spec:
If we were to decide to write a new spec/protocol, what would it look like?
Hereās my rough draft (back of paper napkin idea):
- Feeds are JSON file(s) fetchable by standard HTTP clients over TLS
- WebFinger is used at the root of a userās domain (or multi-user) lookup. e.g:
prologic@mills.io
->https://yarn.mills.io/~prologic.json
- Feeds contain similar metadata that weāre familiar with: Nick, Avatar, Description, etc
- Feed items are signed with a ED25519 private key. That is all āpostsā are cryptographically signed.
- Feed items continue to use content-addressing, but use the full Blake2b Base64 encoded hash.
- Edited feed items produce an āEditedā item so that clients can easily follow Edits.
- Deleted feed items produced a āDeletedā item so that clients can easily delete cached items.
$name$
and then dispatch the hashing or checking to its specific format.
I have submitted this to be used as the hash tooling for Yarn. See it as a good example on using this in a production environment!
Logged in using new argon2i password hash!
$name$
and then dispatch the hashing or checking to its specific format.
Circling back to the IsPreferred method. A hasher can define its own IsPreferred method that will be called to check if the current hash meets the complexity requirements. This is good for updating the password hashes to be more secure over time.
func (p *Passwd) IsPreferred(hash string) bool {
_, algo := p.getAlgo(hash)
if algo != nil && algo == p.d {
// if the algorithm defines its own check for preference.
if ck, ok := algo.(interface{ IsPreferred(string) bool }); ok {
return ck.IsPreferred(hash)
}
return true
}
return false
}
https://github.com/sour-is/go-passwd/blob/main/passwd.go#L62-L74
example: https://github.com/sour-is/go-passwd/blob/main/pkg/argon2/argon2.go#L104-L133
$name$
and then dispatch the hashing or checking to its specific format.
Hold up now, that example hash doesnāt have a
$
prefix!
Well for this there is the option for a hash type to set itself as a fall through if a matching hash doesnāt exist. This is good for legacy password types that donāt follow the convention.
func (p *plainPasswd) ApplyPasswd(passwd *passwd.Passwd) {
passwd.Register("plain", p)
passwd.SetFallthrough(p)
}
https://github.com/sour-is/go-passwd/blob/main/passwd_test.go#L28-L31
$name$
and then dispatch the hashing or checking to its specific format.
Here is an example of usage:
func Example() {
pass := "my_pass"
hash := "my_pass"
pwd := passwd.New(
&unix.MD5{}, // first is preferred type.
&plainPasswd{},
)
_, err := pwd.Passwd(pass, hash)
if err != nil {
fmt.Println("fail: ", err)
}
// Check if we want to update.
if !pwd.IsPreferred(hash) {
newHash, err := pwd.Passwd(pass, "")
if err != nil {
fmt.Println("fail: ", err)
}
fmt.Println("new hash:", newHash)
}
// Output:
// new hash: $1$81ed91e1131a3a5a50d8a68e8ef85fa0
}
This shows how one would set a preferred hashing type and if the current version of ones password is not the preferred type updates it to enhance the security of the hashed password when someone logs in.
https://github.com/sour-is/go-passwd/blob/main/passwd_test.go#L33-L59
I made a thing. Its a multi password type checker. Using the PHC string format we can identify a password hashing format from the prefix $name$
and then dispatch the hashing or checking to its specific format.
Tell me you write go like javascript without telling me you write go like javascript:
import "runtime/debug"
var Commit = func() string {
if info, ok := debug.ReadBuildInfo(); ok {
for _, setting := range info.Settings {
if setting.Key == "vcs.revision" {
return setting.Value
}
}
}
return ""
}()
Esta es una prueba para ver que ocurre con los #hashes en twtxt. (greetings friends)
@movq@www.uninformativ.de, any plans still to clean up the hash from the twtxtās body? Maybe a Festivus gift? You know, āfor the rest of usā. :-D
Cambié el Pwd a Argon2ID, pues es una recomendación como puedes ver acÔ: https://medium.com/analytics-vidhya/password-hashing-pbkdf2-scrypt-bcrypt-and-argon2-e25aaf41598e
@movq@www.uninformativ.de was the request to remove the hash (subject) from showing on twts discarded? I donāt see it on the TODO, so I am curious. Was it something you decided was not worth investing time on?
@movq@www.uninformativ.de, is removing the hash from the body of the twt on the TODO? I read it, but I am unsure if it is there already, or not. š Sorry if it is, and I failed to spot it!
@movq@www.uninformativ.de You can always use a 5GB video file if the UI hashes it with SHA512 before posting to the server.
@movq@www.uninformativ.de
With those two (Message-ID, and In-Reply-To) the hashing could become superfluous, and no longer needed. I would vote for that!
I am noticing that Yarn doesnāt treat āoutsideā (that is, twts coming from a client other than Yarn) twts hashes right. Two examples:
There are many more, but those two will give you the gist. Yarn links the hash to the posterās twtxt.txt, so conversation matching will not work.
So I should really try setting up a Restic repo on an IPFS/IPNS hash.
No, Iām still doing them manually. š¤£š¤¦š» But I do think they are a good idea and will be adding them, I just havenāt gotten around to finding a compatible implementation of the hash yet.
@prologic@twtxt.net @anth Sounds like a good idea. The hash to conv/search url should stay local to a pod.