[PDB Tech] new rate limiting mechanism is too strict

[Chris Caputo]

All,

I am behind the throttling rollout in the last 24 hours, and have worked 
with Theo to loosen things up for now. I've also reached out to pierky re 
changes requested for arouteserver and will endeavor to delay resumption 
of the same throttling until after arouteserver users have had reasonable 
time to upgrade.

Highlights for all client developers:

 - Implement support for PeeringDB API keys:

     https://docs.peeringdb.com/howto/api_keys/

   The idea being that we will throttle users using API keys less.

 - Add a delay in between queries that is randomly between 2 and 2.5 
   seconds, to reduce thundering herd. This delay will mean a client 
   queries PeeringDB at most 30 hits per minute, which will be unthrottled 
   if they are authenticated using API keys and not making identical 
   requests.

 - Highly preferred over separate queries: If you don't need non-public 
   contact info from PeeringDB, is that you implement peeringdb-py 
   (peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side 
   caching. Doing so enables you to locally query the heck out of a local 
   sqlite (or whatever) database. The start time of a peeringdb-py run 
   should be randomized per the docs 
   (http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use 
   peeringdb-py and here is what the once per hour update looks like for 
   all of PeeringDB:

[17/May/2022:15:40:09 +0000] "GET /api/org?since=1652794724&depth=0 HTTP/1.1" 200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423
[17/May/2022:15:40:10 +0000] "GET /api/fac?since=1652773361&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409
[17/May/2022:15:40:10 +0000] "GET /api/net?since=1652796557&depth=0 HTTP/1.1" 200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426
[17/May/2022:15:40:11 +0000] "GET /api/ix?since=1652397370&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397
[17/May/2022:15:40:11 +0000] "GET /api/ixfac?since=1652763759&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414
[17/May/2022:15:40:12 +0000] "GET /api/ixlan?since=1652781160&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399
[17/May/2022:15:40:12 +0000] "GET /api/ixpfx?since=1652429334&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408
[17/May/2022:15:40:13 +0000] "GET /api/netfac?since=1652790428&depth=0 HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553
[17/May/2022:15:40:14 +0000] "GET /api/netixlan?since=1652796556&depth=0 HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590
[17/May/2022:15:40:14 +0000] "GET /api/poc?since=1652785835&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640

   It is fast because, as I understand it, django serializes PeeringDB 
   changes, and the timestamp (since last update) results in only the 
   changes being delivered.

Finally: My apology to those disrupted by this. Please feel free to reach 
out to me with any questions or concerns.

Thanks,
Chris