[PDB Tech] new rate limiting mechanism is too strict

[Theo de Raadt]

At YYCIX, we run arouteserver, which polls peeringdb.  On May 15 (I think)
we started seeing failures to download peeringdb records, which has resulted
in our routeserver configuration not being updated as usual.

I've become aware that the API has a new rate filtering mechanism, for non-APIKEY
accesses.

Well, arouteserver doesn't do APIKEY, and I don't see how the author of
arouteserver would have received any notice that this was suddenly
mandatory, nor all the arouteserver users.

I think peeringdb should have looked more carefully into the no-APIKEY
accesses to determine what downstream effects might occur from this change.

Testing further manually, I observe some really crazy behaviour.

Using a regular browser, I can keep reloading the same record over and over
without hitting any limit.

Using a command-line client (curl or OpenBSD ftp(1)), i hit some extremely
strict limits very quickly.  

On the 11th lookup, I receiving this, and it started counting down:

{"message": "Request was throttled. Expected available in 59 minutes. Authenticate for less restrictions. For more information: https://docs.peeringdb.com/howto/api_keys/", "meta": {"error": "Too Many Requests"}}

All requests are blocked for an hour.

I paused, and tried a new lookup every 5 minutes.  Half an hour later it allowed me
to retrieve 1 record, but then continued the countdown:

{"message": "Request was throttled. Expected available in 17 minutes. Authenticate for less restrictions. For more information: https://docs.peeringdb.com/howto/api_keys/", "meta": {"error": "Too Many Requests"}}

If the new limit is 10 records an hour, I mean you could really just shut
down the service entirely, there's really no difference.

I understand using APIKEY is the new way, but the old way has effectively been
disabled without notice.

But also, why does this work in the browser?  I can reload hundreds of
times.  Is the official browser User-agent being considered exempt from
the rule?  If so I don't understand what the protection plan is.. maybe
we should all change our commandline tools to declare they are Chrome?