00:07 < Savander> "cider": WANT NEW DDOS ATTACK?
00:07 < Savander> on server
00:07 < Savander> hmm
00:11 <@deen> yeah, have been getting some strong attacks today
00:11 <@deen> they look a bit intelligent
00:11 <@deen> since they cause load of 50 and outgoing traffic as well
00:12 < Savander> again, without reason
00:14 < Savander> stupid guys
01:01 < eeeee> i can finish my security patch that prevents connections from spoofed ips without allocating slots on server if you think that'd help
01:04 < eeeee> also how do they cause load of 50 if teeworlds is single threaded? do you have 50 servers running?
01:07 <@deen> eeeee: i think the kernel spawns workers if lots of icmp pings come in
01:08 <@deen> but i may be misremembering
01:09 < eeeee> could add an iptables rule to count icmps
01:09 <@deen> yes
01:09 < o_be_one> well deen we have a problem with change info delay
01:09 <@deen> o_be_one: on block2 ger?
01:09 < o_be_one> yes
01:09 <@deen> It's been fixed for weeks
01:10 <@deen> but that server is never empty
01:10 <@deen> so it never restarts^^
01:10 <@deen> should i restart it now?
01:10 < o_be_one> yes i understand, do you want that i restart it a night ? i'm able to do that cause i live in Canada and when its 10pm for me its 4am for you ... There is only 10 to 20 tees at this hour
01:11 <@deen> well, I'm also awake at 4 am, but last night at 4 am there were 30 players^^
01:11 < o_be_one> yes, normal the weekend
01:11 <@deen> you can shutdown with your account?
01:11 <@deen> then you can do it, sure
01:11 < o_be_one> but during the week you'll have less players
01:11 <@deen> just broadcast about rejoining the server before etc
01:11 <@deen> so players aren't confused
01:11 < o_be_one> i have the admin rcon so i have all commands
01:12 < o_be_one> yes sure
01:12 < o_be_one> just the restart command will do the job ?
01:12 <@deen> shutdown should work
01:12 <@deen> i don't know what restart does^^
01:12 < o_be_one> yes shutdown ok you have check to restart
01:13 <@deen> nah, restart just restarts the game
01:13 <@deen> so it moves everyone to spawn
01:13 <@deen> but doesn't restart the server
01:13 < o_be_one> ok i'll do a shutdown and the system wil start the server again, its ok
01:13 <@deen> yes
01:13 < eeeee> when i was hosting a server i did "restart 60" for the dramatic countdown
01:13 < o_be_one> i'll do it, is it ready ?
01:13 <@deen> just a simple loop to restart the servers
01:13 < eeeee> and then manually "shutdown" when it reaches 0
01:13 <@deen> o_be_one: it's always ready
01:14 < o_be_one> :D
01:14 <@deen> eeeee: cool^^
01:14 < o_be_one> nice
01:14 < o_be_one> thats curious these things, the guy is able to change nick without log in console oO
01:14 < o_be_one> or without notification ingame
01:14 <@deen> yeah, i know how
01:14 <@deen> it's my fault^^
01:14 <@deen> he sends a start packet
01:15 <@deen> and on ddnet servers you can send them even after having started
01:15 <@deen> i think that was necessary for timeout protection stuff
01:15 < o_be_one> funny ^^ ok
01:17 <@deen> in 1 month ddnet will be 2 years old
01:18 <@deen> i need some fun statistics to present then
01:18 <@deen> like: most finishes on a map for a single player, highest time of a finish, etc
01:21 < o_be_one> awesome!
01:21 < eeeee> please also find the biggest team to finish
01:24 <@deen> oh, talking about cool stuff: extreme tw physics abuse: https://youtu.be/JObJvGL_2IA
01:24 <@deen> that was yesterday before the tournament
01:28 < o_be_one> wtf thats amazing :D !
01:29 <@deen> so difficult to coordinate all these tees though^^
01:29 <@deen> we tried with more but always some failed
01:31 < o_be_one> i can imagine ..
08:35 <@EastByte> deen: I'm aware of cider's dos attack
08:37 <@EastByte> eeeee: ping
13:57 < Nimda> 4Nubs by BamBam just released on Novice at 2015-06-16 13:54
15:23 <@deen> EastByte: oh?
15:24 <@EastByte> cider does inforequest flooding using his vps
15:24 <@EastByte> that's the reason for the outgoing traffic
15:25 <@EastByte> and what I think is interesting, simple udp flooding (small packets, huge number of packets per second) causes high cpu usage
16:23 < o_be_one> hello
16:24 < BeaR_> hi
16:25 <@EastByte> hellow
18:46 < o_be_one> omg
18:47 <@EastByte> hm?
18:49 < o_be_one> ah no, its ok ^^
18:52 < Nimda> Heaven Time by Ñı©Ø just released on Solo at 2015-06-16 18:51
19:05 < BeaR_> deen: Is this formatting tool of you working (https://github.com/def-/nimfmt)?
19:05 < BeaR_> working in terms of style guide conform code ouput (:
19:06 <@EastByte> would be promising for 25 lines of code
19:08 < BeaR_> well it seems to use some compiler internals, so maybe
19:08 <@EastByte> indeed
19:11 < BeaR_> and fixing all my wrapper code by hand, would take a while /: (https://github.com/msiglreith/nim-mantle/blob/master/src/mantle.nim)
20:24 <@deen> BeaR_: it's not entirely working. try it out. also the output is not very nice
20:25 <@deen> EastByte: yeah it does. i just checked and even under normal usage (right now) ddnet ger handles 10-20 k udp packets per second
20:26 <@deen> and there#s even a todo at the top, BeaR_ 
20:26 <@deen> BeaR_: i wouldn't spend much time fixing wrapper code style
20:26 <@deen> and hope that someone makes something out of nimfmt instead^^
20:27 <@deen> BeaR_: but good to see a mantle wrapper already =)
20:27 <@deen> people were talking about wanting that
20:32 <@EastByte> deen: I did some stress testing on a local tw server with about 250k packets per second but the gameplay still was smooth
20:32 <@deen> nice
20:32 <@EastByte> although one core seemed to be 100%
20:32 <@deen> so what's with cloudflare saying you can only have 50k packets per second per core? :P
20:33 <@deen> https://blog.cloudflare.com/how-to-receive-a-million-packets/
20:33 <@deen> "While I agree that 50kpps per core is probably the limit for any practical application, what is the Linux networking stack capable of?"
20:33 <@EastByte> ah need to read that
20:34 <@EastByte> the traffic on loopback was at 60mbit/s in and out
20:34 <@EastByte> and I divided it by 20 (packet size)
20:35 <@EastByte> oh wait
20:35 <@deen> anyway, 50k seems like very little
20:36 <@deen> but there is some cool stuff in the article
20:36 <@deen> sendmmsg would be nice for tw i guess
20:38 <@EastByte> "It allows us to send many packets in one go. Let's do 1,024 packets at once." heh
20:39 <@deen> i use software-multi-queuing on the multi-core ddnet servers
20:40 <@deen> never have and hw multi-queue NIC available through virtualization
20:40 <@deen> any*
20:40 <@deen> infers some performance loss, but better than being restricted to 1 core for networking
20:41 <@deen> especially better in ddos situations
20:41 <@EastByte> ah
20:44 <@deen> i think the main thing i saw was sendmmsg that we could use in ddnet server
20:44 < BeaR_> thanks deen, will maybe try it out tomorrow (:
20:45 <@deen> EastByte: especially since ddnet server sends so many packets at once on each tick
20:45 <@EastByte> deen: can we expect to be able to use sendmmsg on avarage vps'?
20:45 <@EastByte> 'added recently'
20:45 <@deen> "The sendmmsg() system call was added in Linux 3.0"
20:45 <@EastByte> oh
20:46 <@EastByte> well probably not available on openvz
20:46 <@deen> meh
20:46 <@deen> would be interesting to try
20:46 <@EastByte> yep
20:46 <@deen> half of my servers are > 3.0
20:46 <@deen> including GER, so worth a try
20:47 <@deen> ddnet brazil is the only x86 (32bit) server 
20:47 <@deen> it sometimes shows unique problems
20:47 <@deen> in a tournament the tuning for gravity was so close to 0 that it had no effect on that 1 server
20:47 <@deen> lots of fun debugging this in a few seconds while people are racing^^
20:47 <@EastByte> haha :D
20:48 <@deen> and they complain that the part doesn't work
20:48 <@deen> float numbers are terrible
20:48 <@EastByte> so about cider's attack, could recvmmsg be used to reduce the cpu load?
20:49 <@deen> wouldn't epoll be better?
20:49 <@EastByte> nah epoll doesn't help in this case
20:49 <@deen> why not?
20:49 <@EastByte> even if you are just while: recv(...) // blocking socket
20:49 <@EastByte> it's still cpu intense
20:49 <@deen> but less so i thought
20:50 <@deen> maybe i'm wrong, experiments should help^^
20:50 <@EastByte> hm didn't see any difference
20:51 <@deen> reducing the number of syscalls is always good :P
20:51 <@EastByte> but if recvmmsg allows us to receive multiple packets at once, it has to be worth it
20:51 <@deen> i saw better performance by caching the timing when log line is written
20:51 <@deen> the time*
20:51 <@EastByte> yep
20:52 <@deen> now the only significant syscalls are indeed recv and send, so yes, try to reduce them =)
20:52 <@EastByte> until now I never thought about pps performance
20:53 <@EastByte> I thought it's more about the traffic caused by huge packets
20:53 <@deen> noooo
20:53 <@deen> the traffic isn't the problem in many attacks
20:53 <@deen> it's easier to overload our cpu
20:53 <@deen> but then some iptables rules would also help
20:53 <@EastByte> yep
20:53 <@deen> but iptables itself is expensive as well
20:54 <@EastByte> not sure how to see cpu load caused by kernel stuff
20:55 <@deen> enable kernel threads in htop?
20:56 <@deen> and press h in htop, the red bar should be kernel
20:56 <@EastByte> oohhh
20:56 <@deen> when a ddos attack happens, it's mostly all red
20:56 <@EastByte> okay that makes sense
20:56 <@deen> measuring the jump from userspace to kernel in syscalls is more difficult
20:56 <@deen> and takes quite long
20:57 <@EastByte> the measuring or the syscall?
20:57 <@deen> the syscall. a syscall has 1-20 µs overhead
20:57 <@deen> while you can do simple stuff in userspace in a few ns
20:58 <@EastByte> well
21:00 <@deen> it's funny to think about: in the time it takes a cpu to add 2 numbers, light only moves sth in the order of 30 cm
21:00 <@EastByte> yep sound crazy
21:02 <@deen> "If a processor operates at 1 gigahertz, a signal can only travel a maximum of about 30 centimetres (1 ft) in a single cycle." https://en.wikipedia.org/wiki/Speed_of_light#Practical_effects_of_finiteness
21:03 <@EastByte> I want superluminal velocity in information transfer :(
21:04 <@deen> sounds like scifi :P
21:04 <@deen> more interesting is stacking computers so close to each other that they can work together efficiently
21:04 <@deen> in a 3d space, and still cooling them well
21:04 <@EastByte> the humanity will probably blow itself blow up itself before reaching such speeds^^
21:05 <@EastByte> -blow itself
21:05 <@deen> faster than light communication would break so much existing physics
21:05 <@deen> i don't expect it
21:06 <@EastByte> I wonder how expensive ~100mbit/s 0byte packet flooding is
21:06 <@EastByte> cider probably has no idea what he is doing
21:07 <@deen> even when quantum information seems to travel at infinite speed it doesn't actually transmit any information
21:07 <@EastByte> since you wouldn't do that on your vps
21:07 <@deen> the limitation of information transmittal at the speed of light seems to be built into the universe
21:07 <@deen> EastByte: 2 € / month?
21:07 <@deen> EastByte: just buy a kimsufi thing and spam
21:08 <@deen> luckily most of the attackers have no idea what they're doing
21:08 <@deen> so much easier to destroy services than keep them running
21:08 <@EastByte> but this kind of attack is noticable at every node
21:09 <@EastByte> hmm
21:09 <@deen> i don't think ovh would care, and if they do, find a hoster that doesn't
21:09 <@deen> afk
21:18 <@EastByte> deen: here is an experiment: request flood -> udp bouncer -> ddnet server
21:18 <@EastByte> this is the bouncer: https://gist.github.com/east/45ada147c3f02566c23f
21:18 <@EastByte> and here htop: http://eastbit.net/priv/16_Jun-15-21_15.png
21:19 <@EastByte> the udp bouncer only passes packets and needs more cpu time than the ddnet server
21:22 <@EastByte> (the packets are 0byte packets)
21:23 <@EastByte> using epoll in the udp bouncer would be more syscalls
21:23 <@EastByte> so I'll try recvmmsg now
21:24 <@deen> all red, mostly your kernel is working ;)
21:25 <@EastByte> hm I'm confused
21:26 <@EastByte> does CPU% 120. include or exclude the kernel?
21:38 <@deen> include
21:38 <@deen> it's not running in a separate kernel thread
21:38 <@deen> instead your regular userspace thread makes a switch into kernel space
21:39 <@EastByte> okay
21:39 < ddnet-commits> [ddnet] def- pushed 1 new commit to DDRace64: http://git.io/vLGy9
21:39 < ddnet-commits> ddnet/DDRace64 85795b3 def: Fix typo
21:40 <@deen> Shiki: so, when will you re-enable auto-record?
21:40 <@deen> people are complaining that it's not working and they lose their valuable recordings
21:40 <@EastByte> well it makes sense, core 3 at 100% is caused by udpbouncer (which causes most packets) while the two other cores have less cpu load
21:48 <@EastByte> looks like one core on my cpu can handle 104mbit/s (20 byte packets)
21:56 <@EastByte> hm I can't find what causes RECORDER_AUTO not to record
21:57 <@EastByte> ah it's about OnStartGame()
21:58 <@EastByte> https://github.com/def-/ddnet/blob/DDRace64/src/game/client/gameclient.cpp#L1081
21:59 <@deen> i guess just change that?
22:00 <@deen> or would it break anything again?
22:00 <@EastByte> not sure yet
22:01 < ddnet-commits> [ddnet] def- pushed 1 new commit to DDRace64: http://git.io/vLGx1
22:01 < ddnet-commits> ddnet/DDRace64 8f4e8a8 def: Make autorecorder work with race again
22:03 <@EastByte> did you test it?
22:03 <@EastByte> not sure why but it doesn't work for me
22:06 <@EastByte> nope the condition is not true
22:06 <@EastByte> has to do with the roundstartick and stuff
22:11 <@EastByte> deen: what actually tried Shiki to fix?
22:12 <@deen> meh
22:12 <@deen> no idea
22:12 <@EastByte> I guess it has to do with vanilla rounds
22:15 <@EastByte> we have three recorders: auto, race and manual
22:15 <@EastByte> auto is ought to record everything
22:15 <@deen> yes
22:15 <@EastByte> so race should be replaced with a 'vanilla' recorder in case of vanilla gaming
22:17 <@EastByte> maybe we should revert the Shiki's update first
22:17 <@deen> please go ahead
22:18 <@EastByte> I guess there are magical git commands for doing that
22:18 <@EastByte> without destroying the commit history ^^
22:20 <@deen> git revert
22:38 < ddnet-commits> [ddnet] east opened pull request #222: Revert: Changes on auto demo recorder (DDRace64...somerevert) http://git.io/vLZY8
22:38 <@EastByte> so this definitely works
22:38 <@deen> ty
22:38 <@EastByte> now I just need to figure out what he fixed
22:38 < ddnet-commits> [ddnet] def- pushed 2 new commits to DDRace64: http://git.io/vLZYu
22:38 < ddnet-commits> ddnet/DDRace64 45748f3 east: Revert: Changes on auto demo recorder
22:38 < ddnet-commits> ddnet/DDRace64 ffa240b Dennis Felsing: Merge pull request #222 from east/somerevert...