19:50 <@minus> i just set this baby up: https://sr.ht/ZtRE.png
19:50 <@minus> and already see it showing ugly stuff
19:53 < deen> minus: as you may know the master servers have been getting ddosed for a few days or weeks
19:54 <@minus> DoSed*
19:54 < deen> right, my bad
19:54 <@minus> yeah
19:55 <@minus> i do have an idea how to make it better
19:55 <@minus> i guess i could just try hacking around because it can't get worse than it is
19:58 < deen> what system are you using for monitoring btw?
19:58 < deen> I know I've seen it somewhere, but forgot the name
19:58 <@minus> that's grafana
19:58 <@minus> i use prometheus as backend
19:59 < deen> you can also see the reason for 0 players here: https://ddnet.tw/stats/server/#ger.ddnet.tw
19:59 <@minus> i can?
20:00 < deen> master4 is on GER
20:00 <@minus> oh, there's a spike in incoming traffic
20:00 <@minus> mastersrv does burn through CPU atm, mhm
20:01 <@minus> have you sampled the attack traffic?
20:01 < deen> sure
20:02 < deen> we have a few different kinds of attacks against ddnet, not just the mastersrv one, so not every spike is a mastersrv attack though
20:04 < EastByte> like I already stated, the masters are flooded with heartbeat packets 'b', 'e', 'a', '2' and that leads to an overflow in the serverlist
20:04 < deen> oh?
20:04 < EastByte> the problem could be solved doing master-side serverlist caching
20:04 < deen> first time i hear that
20:05 < deen> sounds solvable
20:05 <@minus> no idea what you mean by caching
20:05 <@minus> but it could be solved by preferring older servers over newer servers
20:05 < deen> "solved" in the sense that they have to keep the attack up indefinitely for it to become useful
20:06 <@minus> i guess
20:07 <@minus> though keeping up a 10mbit/s stream of data indefinitely doesn't sound hard
20:07 < deen> right, can get that for 5 € / month with ip spoofing
20:08 < EastByte> minus: that's basically what I mean
20:09 < EastByte> but I guess the master should completely ignore the these packets on flooding
20:10 < EastByte> s/the these/these/
20:10 < EastByte> (from unknown sources)
20:13 < EastByte> we could also manually manage serverlists or set up some kind of webpanel for server registration
20:13 < deen> please no..
20:15 < EastByte> well the mastsrv protocol is vulnerable and we can't change that without mitigating to a new teeworlds version
20:18 <@minus> those 70 servers that are registered, are those yours, deen?
20:18 <@minus> then again no
20:19 < EastByte> those servers are registered on the same server as master4
20:19 <@minus> oh, they are
20:20 <@minus> are you blocking all heartbeats, deen?
20:20 <@heinrich5991> minus: could you replace the URL http://teeworlds.com/ with https://www.teeworlds.com/ on https://github.com/teeworlds/teeworlds
20:21 < deen> minus: yeah, i tried to block just the bad ones, but don't see a way to tell the difference and I hear that the incoming traffic (or cpu from mastersrv) cause lags
20:21 <@minus> yeah, fair
20:24 < EastByte> minus: have you looked at the mastersrv's console output?
20:24 <@minus> no
20:25 <@minus> why?
20:25 <@minus> i know it's lots
20:26 < EastByte> because it should say 'error: mastersrv is full'
20:27 < EastByte> but the last time I checked it seems like the other master servers didn't response at all while the attack was going on
20:28 < EastByte> so I guess the hoster's anti-ddos system dropped it all
20:42 <@minus> deen: it stopped, you can deactivate the rule
20:43 < deen> minus: you're lying!
20:44 < deen> I just tried and it's still ongoing
20:44 <@minus> https://sr.ht/9HyI.png
20:45 < deen> yeah, but our server is still under attack
20:45 <@minus> well, that sucks
20:45 <@matricks> :/
20:46 < deen> we also have another problem right now, some guy running around banning everyone on every server
20:46 <@minus> but dat graph doe
20:46 <@minus> votekicking or literally banning
20:46 < deen> votekick and he has a horde of dummies
20:47 < deen> so every vote goes through
20:47 <@matricks> if I ever do a multiplayer game again.. kick me
20:51 <@heinrich5991> ah, someone finally gathered enough IP addresses to do that?
20:51 <@heinrich5991> do they remain static, so we could ban them?
20:53 <@minus> i wonder if anyone runs ELK or graylog on their teeworlds logs
20:53 <@heinrich5991> what's that?
20:53 <@minus> log processing and indexing stacks
20:54 <@minus> making logs searchable and extracting data from it
20:56 < deen> heinrich5991: ask EastByte, he's looking into it
20:57 < deen> and find a solution please^^
21:06 < koomi> minus: maybe try not to appeal to 12 year-olds next time
21:06 <@minus> *matricks?
21:06 < koomi> er, yes
21:06 <@matricks> koomi: ages does matter, twats comes in all ages
21:06 <@minus> koomi: you don't want to provide a mastersrv anymore?
21:07 < koomi> minus: I forgot about it until I read about the dos'ing
21:07 < koomi> I can give you a login in a few minutes if you want
21:09 <@minus> oh, there the attack is again
21:10 <@minus> would be nice if you don't mind those attacks (they do eat a lot of CPU after all)
21:24 < needs1> How can a Dos on masters lower servers count?
21:24 <@minus> they flood the checker slots afaiu
21:26 < needs1> Does they flood masters with heartbeats, taking available slots as soon as they are free?
21:27 <@minus> yep
21:30 <@minus> meh, preventing that does require changing how the master works quite a bit
21:30 < needs1> And I guess you can't really ban IPs since it is only one packet without any context, so IPs can be spoofed?
21:30 <@minus> yes
21:31 < needs1> That suck :(
21:37 < needs1> OVH have a good dos mitigating system as far as I know
21:37 < deen> needs1: doesn't help if there is no way to tell the difference between legitimate and DoS traffic
21:38 < needs1> Ah, indeed
22:07 < BotoX> could check if there is a server on that IP
22:08 < BotoX> async ofc and only add if it succeeded (like 5 sec timeout?)
22:08 < BotoX> attacker could fake that too though if they know that the master checks, doesn't have any sequence does it?
22:09 < EastByte> it doesn't
22:09 < BotoX> meh honestly master should've been using TCP
22:10 < BotoX> could send from a few IPs and ban the IP if it responds to the wrong one
22:10 < BotoX> (send the info packet to the server from a pool of IPs)
22:10 < BotoX> then the attacker can't know where it came from
22:11 < BotoX> if he responds to all IPs that he has gathered then he will be banned
22:11 < BotoX> he could try a random one but would quickly get banned anyways
22:11 < BotoX> or he would fill up memory with all the spoofed IPs lel
22:11 < BotoX> and maybe even murder legit servers 
22:11 < EastByte> way too hacky for my taste :p
22:11 < BotoX> well suggest something better heh
22:11 < BotoX> everything is a hack at this point, isn't it?
22:11 < EastByte> yes
22:13 < BotoX> doesn't OVH have ingress filtering to filter spoofed IPs?
22:13 < BotoX> (I think they don't if the spoofed packets come from inside the same DC or something like that?)
22:13 < EastByte> if the source ip is part of the ovh network it will be blocked
22:14 < BotoX> well ingress filtering should work for all networks as far as I am aware?
22:15 < EastByte> how is that supposed to work?
22:15 <@minus> BotoX │ could check if there is a server on that IP ← that's what the master server does
22:15 < BotoX> EastByte: https://en.wikipedia.org/wiki/Ingress_filtering
22:15 < BotoX> By knowing if the IP is routed through that ASN
22:15 < BotoX> minus: so I guess he spoofes that too, right?
22:18 <@minus> no
22:18 < BotoX> so why is it not being ignored?
22:18 <@minus> the attack just occupies all checker slots
22:18 < BotoX> oh I see
22:18 <@minus> it literally maxes the CPU
22:18 < EastByte> BotoX: well, it could work if all network operators over the world would cooperate, which is not the case
22:19 < BotoX> you usually know which ASN you are peering with on which port.....
22:19 < BotoX> since you are doing BGP with them
22:19 < BotoX> (which I hope you are filtering to only their subnets)
22:19 < BotoX> I guess impossible for single homed ISPs
22:22 < koomi> doesn't work for transit peers in general
22:22 < BotoX> yeah ofc. but most bigger DCs peer at exchanges
22:22 < BotoX> so you could partly fix this problem
22:22 < BotoX> of course you'd always have lots of transit traffic
23:56 <@minus> https://sr.ht/wYmk.png how nicely the server count on different masters converges :D