13:35 < lvgx> so what's the issue with master servers?
17:06 < Dune> heinrich5991: was there a git bot at some point here?
18:37 < Dune> Oy merging the pull requests 
18:37 < Dune> \o/
18:41 < Dune> heinrich5991: it would be very nice to have a way to contact Oy and request to be reviewer or assignee for an issue :¹
18:42 < Dune> right now the only place to ask for this stuff would be... the tw.com forums?
18:42 < Dune> I don't know if some subforum could be used for that or something
18:42 < Dune> it's hard for different contributors to coordinate each otehr
20:45 < Dune> minus: hi?
20:50 < minus> hi?
20:50 < Dune> ho
20:50 < Dune> https://www.teeworlds.com/forum/viewtopic.php?pid=121807#p121807
20:50 < Dune> the responsible for the DDos attack is using it to advantage his own servers
20:50 < Dune> do you deem wise to take any action from the mastersrv?
20:59 < minus> basically i don't care, but you can ban them, heinrich5991 has the necessary permissions
21:11 < Dune> aight
21:11 < Marcel> hi
21:11 < Dune> hi Marcel
21:23 < Marcel> soo is there anyone doing something against the master server problem
21:23 < Dune> releasing 0.6.5 and 0.7.0, for one
21:23 < Dune> patching vulnerabilities
21:23 < Marcel> what did you change
21:24 < Dune> https://nvd.nist.gov/vuln/detail/CVE-2018-18541
21:24 < Dune> "In Teeworlds before 0.6.5, connection packets could be forged. There was no challenge-response involved in the connection build up. A remote attacker could send connection packets from a spoofed IP address and occupy all server slots, or even use them for a reflection attack using map download packets."
21:25 < Marcel> and what about the ddos problem
21:25 < Dune> that is the ddos problem
21:25 < Dune> another attack angle is the server info packets
21:26 < Dune> but 0.6.5 fixes the vulnerabilities at least partially, iirc
21:26 < Dune> upgrade your servers (and clients).
21:27 < Marcel> they still could send tons of those packets and by that overload the master server
21:27 < Marcel> the master server right now is not working at all
21:28 < Dune> upgrade to 0.7
21:28 < Dune> masterserver works pretty well there
21:30 < Marcel> well it probably won't take long until they're getting ddosed
21:30 < minus> huh, was 0.7 released?
21:30 < Dune> yes minus
21:30 < Dune> a week ago
21:30 < minus> no one updated the website tho
21:30 < Dune> yeah
21:30 < minus> still 0.6.5 there
21:30 < Dune> take it as a public beta, 0.7.1 should come soon
21:30 < Marcel> the download is still not available tho
21:30 < Dune> you can get it on the github releases
21:30 < Dune> https://github.com/teeworlds/teeworlds/releases
21:31 < Marcel> i suggest to move to http and then use cloudflare or something similar as a protection layer
21:31 < Marcel> by that you also could add new ones whenever you like
21:32 < Dune> I believe the main issue arises from the reflection exploit
21:32 < Dune> but I'm afriad I'm not competent enough to answer you on that
21:32 < minus> https://sr.ht/_I86.png that kind of looks wrong
21:32 < Marcel> well the main servers are completly down (in 0.6) so i think that there's probably more than that
21:33 < Dune> I'm not well in touch with the technicalities of the exploit
21:33 < Dune> minus: is that all you get?
21:33 < minus> yes
21:34 < minus> i see a bit more if i go fullscreen 
21:34  * Marcel slaps heinrich5991 around a bit with a large fishbot
21:34 < minus> could be due to wayland
21:34 < Dune> heinrich is away for now
22:32 <@heinrich5991> minus: is that 0.7?
22:32 < minus> yes
22:32 <@heinrich5991> Marcel: yes, moving to https for master servers is the plan
22:32 <@heinrich5991> :(
22:35 <@heinrich5991> Dune: we can ban their servers from the masters if you want to go forward with that
22:41 < Dune> what protections does 0.6.5 add?
22:41 < Dune> I mean, there are several problems on 0.6.4 right?
22:42 <@heinrich5991> yes. on a "non-technical" level: filling up slots using spoofed IP addresses
22:42 <@heinrich5991> that's what 0.6.5 fixes
22:42 < Dune> so nothing ddos related
22:42 < Dune> what about 0.7.0, on a security level?
22:45 <@heinrich5991> it fixes reflection attacks using spoofed IP addresses
22:45 <@heinrich5991> ddos of masters/servers is still possible using serverbrowse packets
22:46 < Dune> :/
22:54 < Dune> 8[22:32]	heinrich5991: Marcel: yes, moving to https for master servers is the plan
22:55 < Marcel> so what's your current state
22:58 <@heinrich5991> Marcel: had some working parts already
22:59 <@heinrich5991> do you have some experience with this? I'd love to have someone to discuss this with :)
23:00 < Marcel> yea, got some. what language are you using server side
23:02 <@heinrich5991> so far I just used a simple python http end point for the registering and a static file (regularly updated and served by nginx behind cloudflare) for the server list
23:05 < Learath2> There is a huge flood of cou2 packets to master4.teeworlds.com which makes it pretty much toast. Now is it a fault of the protocol I'm not sure as siz2 packets are small enough, any challange response scheme would require a much bigger challange packet
23:06 <@heinrich5991> 0.6.5 challenge packet is huge
23:06 <@heinrich5991> (512 bytes)
23:06 <@heinrich5991> 0.7.0 not so much IIRC
23:06 <@heinrich5991> would require breaking the protocol again ^
23:06 <@heinrich5991> ^
23:06 < Marcel> and does the http request on the client side already work?
23:06 < Learath2> 0.6.5 one is the client challange though right?
23:06 <@heinrich5991> Marcel: yes
23:06 < Learath2> I meant for the masterserver
23:06 <@heinrich5991> Learath2: not needed for http masters
23:06 < Learath2> Yeah but that's a lot of work to get working
23:07 < Marcel> how do the master servers communicate with each other?
23:07 < Marcel> right now
23:07 <@heinrich5991> missing parts are proper server registration on the game server side (and also a strategy for this) and a legacy server scraper to make the old servers compatible with the new system
23:07 < Learath2> They currently don't iirc
23:07 <@heinrich5991> Marcel: currently not at all, clients fetch all server lists, game servers register at one
23:07 <@heinrich5991> I want to change it (as matricks has advised) to game servers register at all masters, clients fetch one
23:08 < Marcel> hmm
23:09 <@heinrich5991> what do you think about that?
23:10 < Learath2> I don't think it's the best idea tbh, it's very possible to knock out a master with the weak servers we have
23:10 < Marcel> what about having a sql server and the master servers syncing with it maybe every 30 seconds
23:11 < Learath2> It centralizes the system but maybe light caching servers that report back to a main server. Server goes through all masters until it finds one. Client does the same
23:12 <@heinrich5991> Learath2: that's why we put the masters behind cloudflare
23:12 <@heinrich5991> if a master is knocked out, the client tries the next one
23:12 < Marcel> i don't think that the client should know all the master servers. i think that it'd be better if there's one address and that cloudflare at the end decides which server to choose (i think they have a service for that)
23:13 <@heinrich5991> I don't want to make cloudflare mandatory for teeworlds
23:13 < Learath2> Yes but we still have the original issue, you can knockout a master and the servers on it are practically unavailable
23:13 <@heinrich5991> they could go out of business one day
23:13 < Marcel> they probably won't, they are so big and still growing
23:13 < Marcel> cloudflare has a really good anti ddos protection and caching system
23:13 <@heinrich5991> yes. but this is an open-source game, it shouldn't depend on such services by design IMO
23:13 <@heinrich5991> yes
23:14 <@heinrich5991> I want to use them
23:14 <@heinrich5991> but I don't want to make another implementation impossible
23:14 < Learath2> We could run a load balancing public facing server. Don't need cloudflare for that anyone provides that
23:14 <@heinrich5991> so putting the masters behind cloudflare is a thing I want to do
23:15 < Learath2> heinrich5991: why do you think it's better for the servers to register at all masters?
23:15 <@heinrich5991> so masters are independent
23:15 <@heinrich5991> and if one isn't reachable by a client, it's still okay
23:16 < Marcel> they still'd be independent with a sql server
23:16 < Learath2> no they wouldn't be, they'd be dependant on a central sql server
23:16 < Dune> i'll go to sleep now, my people need me
23:16 <@heinrich5991> good night Dune :)
23:16 < Learath2> night Dune :)
23:16 < Dune> i hope you'll work out a layer of protection for the masterservers
23:16 <@heinrich5991> nice to see you here agiain
23:16 < Dune> sorry I can't help much
23:17 < Dune> good night : )
23:17 < Marcel> yep, but if the sql server is hidden then it also wouldn't get ddosed
23:17 < Dune> nice to see all that activity, yeah
23:17 < Dune> ;-)
23:17 < Marcel> gn
23:17 <@heinrich5991> okay, but if all game servers register with all masters, we don't need a single point of failure, do we?
23:17 < Learath2> Marcel: no need for it though, it creates unnecessary complexity. I think I'm okay with servers registering at every master
23:18 < Marcel> the problem with that is that the master servers are currently hardcoded and there's no option to add additional ones
23:18 < Learath2> We should let the client fetch from as many masters as it needs though. So that people can run their own e.g.
23:19 < Learath2> Well we could run lightweight proxies on master{1,4}.teeworlds.com to translate to the new system until the legacy usage drops below a certain %
23:19 < rand> what about DHT ? *flying away*
23:20 < rand> http will be nice
23:20 < Marcel> that's a good idea
23:20 < Learath2> Heh think matricks suggested DHT once upon a time
23:21 <@heinrich5991> that's not a solution that will work in one week
23:21 <@heinrich5991> we'd better use one that might work in one week
23:21 < Learath2> heinrich5991: so how far are you anyways? anything we can help with?
23:21 <@heinrich5991> well, that was based on ddnet back then
23:22 <@heinrich5991> so currently I'd need to add the curl library to teeworlds 0.6.5, with bam and cmake
23:22 < Learath2> oy isn't even here, does he even accept prs now?
23:23 < rand> small one at least
23:23 < rand> for 0.7
23:23 <@heinrich5991> it seems that I'm kinda in charge of 0.6.5 right now
23:23 <@heinrich5991> I would port it to 0.7 afterwards, obviously
23:24 <@heinrich5991> (after maybe adding cmake to 0.7)
23:24 < rand> :D that would be awesome
23:25 < minus> why cmake when you can have meson
23:25 < rand> what is meson ?
23:26 <@heinrich5991> minus: because I know cmake
23:26 <@heinrich5991> it seems industry-standard, and it works pretty well for ddnet
23:26 < minus> you can learn meson :D
23:26 <@heinrich5991> we do cross-compilation to everything from linux
23:26 < minus> cmake works, yes
23:26 <@heinrich5991> it's pretty awesome
23:26 < minus> it does not work well
23:27 <@heinrich5991> what do you mean? the result or that you have to code in a language that you don't like?
23:27 <@heinrich5991> the cmakelists I wrote for ddnet seem to work pretty well
23:27 <@heinrich5991> I don't like the language, but that's okay for the result
23:27 < Marcel> what about if the client does a request to masters.teeworlds.com and then fetches all the servers from the master servers listed there
23:27 < Learath2> We even have tests nowadays \o/
23:28 <@heinrich5991> yes. I don't think anyone but me has added tests(?)
23:28 < minus> i find it hard to impossible to produce static builds  with it, and example programs of libraries produced in the same project
23:28 <@heinrich5991> Marcel: so clients should connect to multiple masters you say?
23:28 < Learath2> heinrich5991: I did in a couple branches of mine that never got any review so I gave up on them :P
23:28 < rand> Marcel: what if masters.teeworlds.com  got ddos ?
23:29 < Marcel> that's the problem
23:29 < Learath2> Marcel: another single point of failure that's not necessary
23:30 < Learath2> We should instead use masters.cfg. (not the way we have now where it just resets it for some reason)
23:30 < rand> a solution such that it's easier to get a ddos protection is fine, i guess that the threat is weaker as long as it's possible
23:30 < rand> that's why http alone is fine, with or without cloudfare
23:30 < Learath2> masters.cfg initialised with master{1,4}.teeworlds.com; client instead gets the list from one place through http
23:31 < Learath2> server also goes through masters.cfg to register at all masters and heartbeats through http
23:31 < Learath2> (maybe also add a PoW requirement for registering or atleast a challange/response)
23:34 < Marcel> what if the client would do a request to masters.teeworlds.com if the masters.cfg is missing and write them to the file. if the website was not accessable then it could use the hardcoded ones
23:34 < Learath2> What does that gain us?
23:35 < Marcel> the ability to expand
23:35 <@heinrich5991> the ability to expand to what? to more than 4 master servers?
23:36 <@heinrich5991> 4 master servers should be enough for anyone – bill gates
23:36 < Marcel> it's always better to have the chance than forcing everyone to update their clients and servers
23:36 <@heinrich5991> you also add untested code paths
23:37 <@heinrich5991> so that's not a "pure" upside
23:37 <@heinrich5991> we haven't seen a reason to have more than 4 masters in 10 years
23:37 <@heinrich5991> do you think that will change rather than something else we did not foresee?
23:38 < Learath2> heinrich5991: should we maybe add the version to the url btw? So further updates are relatively easier?
23:40 < Marcel> that's just something i've learned in the past few years. products should be prepared for their growth, you also thought that udp'd be good enough.
23:40 <@heinrich5991> I doubt that anyone who thought about that thought that
23:41 <@heinrich5991> we've had a relevant issue open on github for years
23:41 <@heinrich5991> the problem is that you need to be aware of reflection attacks et cetera
23:41 <@heinrich5991> if you aren't adding random things like "having the possibility to add more udp masters" adds nothing
23:46 < Marcel> matricks also probably thought that it'd good enough to write the system as it is now. we can see now what we've got by that. but whatever if you are sure about that
23:47 <@heinrich5991> do you think that this was designed by someone who was aware of reflection attacks?
23:48 < Marcel> i don't know, it could also be that he didn't thought anyone would abuse it
23:53 < Marcel> do you want to write the masters in python? i could write them in php if you want
23:54 <@heinrich5991> personally, I'd probably prefer python
23:54 <@heinrich5991> I like the language better, and I think not only because I know it better
23:55 < Learath2> What will you use as the http library? flask?
23:56 < Marcel> the is probably true, but php is still being widely used and is running on almost every website
23:56 < kebron> What about golang? Would probably be faster
23:58 <@heinrich5991> kebron: the good thing is: you can replace it, it's running on the master
23:58 <@heinrich5991> a quick, working PoC is fine for the start
23:58 < kebron> Yeah true
23:58 <@heinrich5991> otherwise you could indeed rewrite it in go, or even rust :P
23:59 <@heinrich5991> but we don't have to release a new version for that, because it's on our side, not the client's
23:59 < rand> as long as the protocol is well defined :}
23:59 <@heinrich5991> yes
23:59 < rand> whatever the language