@prologic I haven't been tracking these changes or conversation. Can you link me to something so that I can catch up?

@bender I'm hoping my little one makes it that long; it's possible! Unless some medical miracle occurs I'll be long ago converted to dust by then.

@New_scientist No for f*#$'s sake stop pushing this hype

@bender Given that I haven't posted in so long, my lines of twtxt per unit time average is probably lower than most! I'm a bursty twtxter.

@New_scientist No they don't.

@New_scientist Hallucinating invisible pedestrians, great application!

@Phys_org ...which will be entirely ignored when the 💩 hits the 🪭

"Interest grows in geoengineering" because pursuing the obvious, clearest, most direct solution--reducing fossil fuel use--is for some reason off the table. That is already an unethical arrangement. Pasting an ethical framework on top doesn't change the rotten situation at its core.

@New_scientist Pusher can detect own product. Film at 11.

@New_scientist Make sense--if a clown murders the child they don't need to go to the hospital.

@quark wow everybody loves @prologic

@quark Check out this thread if you haven't already: https://mastodon.social/@sundogplanets/112464533481477428

I think we already know It's likely to be a disaster.

@New_scientist It's great that US regulators have approved launching 40,000 satellites with a 5-year lifespan before we had this kind of information about what's likely to happen when they start falling out of orbit at a rate of several per hour.

@prologic My pod, which is running the same commit you are, does not return an error like that. It returns the same HTML it always has. Try it. I nuked my cache before restarting.

Edit: Oh wait, the plot thickens. I do get an error if I use curl or if I use a web browser that isn't logged in. That's good!

@prologic I'm not sure what this update does, but

https://twtxt.net/external?uri=https://google.com&nick=lovetocode999

still exhibits the same problem, on your pod and on mine, after the latest update.

@prologic OK, I just updated to commit 77d527, which looks to be the same one you're running right now. I forgot to blow away my cache before restarting, so I just deleted the cache file and restarted.

@bender Hey, want to go in halfsies on one?

@bender Oh look at that, the same problem is still happening on twtxt.net too. I tested a different link but that one gave an error. Maybe that means my pod isn't behaving different from twtxt.net after all.

A stopgap setting that would let me stop all calls to /external matching a particular pattern (like this damn lovetocode999 nick) would do the job. Given the potential for abuse of that endpoint, having more moderation control over what it can do is probably a good idea.

@prologic I deleted a file named cache in my yarn data and restarted. Problem persists.

@prologic "Refresh cache in Poderator Settings"

Is there some other way to do that?

@prologic What? I compiled, updated, and restarted. If you check what my pod reports, it gives that 7a... SHA. I don't know what that other screenshot is showing but it seems to be out of date. That was the SHA I was running before this update.

@prologic Here's a log entry:

Aug 27 15:59:43 buc yarnd[1200580]: [yarnd] 2024/08/27 15:59:43 (IP_REDACTED) "GET /external?nick=lovetocode999&uri=https://URL_REDACTED HTTP/1.1" 200 35442 14.554763ms

HTTP 200 status, not 404.

@prologic This does not seem to fix the problem for me, or I've done something wrong. I did the following:

1. Pull the latest version from git (I have commit 7ad848, same as on twtxt.net I believe).
2. make build and make install
3. Restart yarnd
4. Refresh cache in Poderator Settings

Yet I still see these bogus /external things on my pod when I hit URLs like the one I sent you recently. When I hit such a URL with curl I think it's giving an error? But in a web browser, the (buggy) response is the same as it was before I updated.

So, this problem is not fixed for me.

@prologic Aha, now it gives an error. OK I'm updating to this to see if it fixes the issue on my pod! Thank you.

@prologic I believe you are not seeing the problem I am describing.

Hit this URL in your web browser:

https://twtxt.net/external?nick=lovetocode999&uri=https://socialmphl.com/story19510368/doujin

That's your pod. I assume you don't have a user named lovetocode999 on your pod. Yet that URL returns HTTP status 200, and generates HTML, complete with a link to https://socialmphl.com/story19510368/doujin, which is not a twtxt feed (that's where the twtxt.txt link goes if you click it). That link could be to anything, including porn, criminal stuff, etc, and it will appear to be coming from your twtxt.net domain.

What I am saying is that this is a bug. If there is no user lovetocode999 on the pod, hitting this URL should not return HTTP 200 status, and it should definitely not be generating valid HTML with links in it.

Edit: Oops, I misunderstood the purpose of this /external endpoint. Still, since the uri is not a yarn pod, let alone one with a user named lovetocode999 on it, I stand by the belief that URLs like this should be be generating valid HTML with links to unknown sites. Shouldn't it be possible to construct a valid target URL from the nick and uri instead of using the pod's /external endpoint?

@prologic @bender I partially agree with bender on this one I think. The way this person is abusing the /external endpoint on my pod seems to be to generate legitimate-looking HTML content for external sites, using a username that does not exist on my pod. One "semantically correct" thing to do would be to error out if that username does not exist on the pod. It's not unlike having a mail server configured as an open relay at this point.

It would also be very helpful to give the pod administrator control over what's being fetched this way. I don't want people using my pod to redirect porn sites or whatever. If I could have something as simple as the ability to blacklist URLs that'd already help.

@lyse Interesting. The yarnd --help currently says (for me):

  -R, --open-registrations            whether or not to have open user registgration

meaning it doesn't give the default setting or warn you that you need to use -R=false and not -R false. It also leaves unclear whether --open-registrations false would work or if you need to do --open-registrations=false. It's also unclear whether the setting change in the user interface is overridden by the command line arguments, overrides the command line arguments, is persisted across restarts.

Maybe all this is worth posting an issue for additional documentation on the git repo if there isn't one already.

"registgration" is misspelled that way in the help by the way.

@lyse Ha, sweet thanks for this! For some reason I thought you had to do this with an environmental variable or command-line option and I didn't think to check the settings. 🤦‍♂

@prologic Ah nice, thank you! Do you think this fix is ready for me to test it or do you think I should wait til you poke at it?

For some reason this nick lovetocode999 is frequently present in my log entries.

@mckinley He's signed up three times now even though I keep deleting the account, which is enough for me to permaban this person. I don't technically want open registrations on my pod but up till now I've been too lazy to figure out how to turn them off and actually do that, and there hasn't been a pressing need. I may have to now.

@support No. Try this again and I nuke your IP.

@support Nope.

@prologic I don't think it's your code. As you said in one of your commit comments, the internet is a hostile place! That's partly why I reacted the way I did: all things considered it's usually better to react quickly and clean up the mess later, then it is to wait and risk further damage. Anyway it sucks @xuu got caught up in it. Hopefully it's all good now.

@xuu I hope everything is sorted out with your ISP. Please let me know if there's anything I can do to help. I sincerely did not mean to cause you any trouble.

@stigatle @xuu @lyse "Not cool"? I was receiving many broken (HTTP 400 error) requests per second from an IP address I didn't recognize, right after having my VPS crash because the hard drive filled up with bogus data. None of this had happened on this VPS before, so it was a new problem that I didn't understand and I took immediate action to get it under control. Of course I reported the IP address to its abuse email. That's a 100% normal, natural, and "cool" thing to do in such a situation. At the time I had no idea it was @xuu .

The moment I realized it was @xuu and definitely a false alarm, I emailed the ISP and told them this was a false positive and to not ban or block the IP in question because it was not abusive traffic. They haven't yet responded but I do hope they've stopped taking action, and if there's anything else I can do to certify to them that this is not abuse then I will do that.

I run numerous services on that VPS that I rely on, and I spent most of my day today cleaning up the mess all this has caused. I get that this caused @xuu a lot of stress and I'm sincerely sorry about that and am doing what I can to rectify the situation. But calling me "not cool" isn't necessary. This was an unfortunate situation that we're trying to make right and there's no need for criticizing anyone.

@prologic Yes, I can do that.

@stigatle @prologic my /tmp is also fine now! Thanks for your help @prologic!

@prologic I unbanned a few IP address I had blocked before the bugfix. I wasn't being careful and just blocked any IP I saw making a large number of requests to my pod. That slowed the problem down but I think I blocked your and @stigatle 's pods in the process, oops.

@stigatle Sweet, thank you! I've been shooting myself in the foot over here and want to make sure the situation is getting fixed!

@stigatle @prologic testing 1 2 3 can either of you see this?

@prologic I don't know if this is new, but I'm seeing:

Jul 25 16:01:17 buc yarnd[1921547]: time="2024-07-25T16:01:17Z" level=error msg="https://yarn.stigatle.no/user/stigatle/twtxt.txt: client.Do fail: Get \"https://yarn.stigatle.no/user/stigatle/twtxt.txt\": dial tcp 185.97.32.18:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" error="Get \"https://yarn.stigatle.no/user/stigatle/twtxt.txt\": dial tcp 185.97.32.18:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"

I no longer see twts from @stigatle at all.

@prologic Have you been seeing any of my replies?

It shows up in my twtxt feed so that's good.

@prologic

./tools/dump_cache.sh: line 8: bat: command not found
No Token Provided

I don't have bat on my VPS and there is no package for installing it. Is cat a reasonable alternate?

@prologic Try hitting this URL:

https://twtxt.net/external?nick=nosuchuser&uri=https://foo.com

Change nosuchuser to any phrase at all.

If you hit https://twtxt.net/external?nick=nosuchuser , you're given an error. If you hit that URL above with the uri parameter, you can a legitimate-looking page. I think that is a bug.

@prologic Hitting that URL returns a bunch of HTML even though there is no user named lovetocode999 on my pod. I think it should 404, and maybe with a delay, to discourage whatever this abuse is. Basically this can be used to DDoS a pod by forcing it to generate a hunch of HTML just by doing a bogus GET like this.

I'm seeing GETs like this over and over again:

"GET /external?nick=lovetocode999&uri=https://vuf.minagricultura.gov.co/Lists/Informacin%20Servicios%20Web/DispForm.aspx?ID=8375144 HTTP/1.1" 200 35861 17.077914ms

always to nick=lovetocode999, but with different uris. What are these calls?

@stigatle I used the following hack to keep my VPS from running out of space: watch -n 60 rm -rf /tmp/yarn-avatar-*, run in tmux so it keeps running.

The vast majority of this traffic was coming from a single IP address. I blocked that IP on my VPS, and I sent an abuse report to the abuse email of the service provider. That ought to slow it down, but the vulnerability persists and I'm still getting traffic from other IPs that seem to be doing the same thing.