FishGuy876 (03:29:44)
I need another beer hehe

FishGuy876 (03:28:54)
It most likely figured that out and just kept sending new requests on a slightly different address. A lot of these shitty crawlers do not respect the rules or the robot.txt

FishGuy876 (03:28:24)
The problem is this ancient django. That crawler was using a slightly different ipv6 address every time, so the limiter even on cloudflare didnt catch it (it blocks if more than 10 requests per second occur on an address)

LimeProselyte (03:27:42)
like a mini DDoS

LimeProselyte (03:27:38)
50 reqs a second is insane

LimeProselyte (03:27:28)
at some point you might be able to add a rate limiter to Django

FishGuy876 (03:21:24)
100% of the problem will then be mostly gone, as i'll put the tag pages in the same category

FishGuy876 (03:20:58)
arabek gave me a great idea bit ago for some of those types of problems. So later this week, i'll make a code change that a user must be logged in to view a song page

FishGuy876 (03:20:16)
So not a bad test really

FishGuy876 (03:20:10)
It was hitting the server at around 50 requests a second, for the last 30 minutes, all at uncached pages on tags. Server did OK, it was just maxxed out on the workers serving those requests, and didnt fall over completely

Klassiko

Captain Tortilla (Dennis Mott)

djrandom

Time Left: 2:04