[PDB Tech] Fwd: new rate limiting mechanism is too strict

Thu Apr 11 11:28:12 UTC 2024

fyi, Arnold

-------- Weitergeleitete Nachricht --------
Betreff: Re: [PDB Tech] new rate limiting mechanism is too strict
Datum: Tue, 17 May 2022 16:54:59 +0000 (UTC)
Von: Chris Caputo <ccaputo at alt.net>
Antwort an: Chris Caputo <ccaputo-dated-1663174500.27c315 at alt.net>
An: pdb-tech at lists.peeringdb.com
Kopie (CC): Theo de Raadt <deraadt at yycix.ca>

All,

I am behind the throttling rollout in the last 24 hours, and have worked 
with Theo to loosen things up for now. I've also reached out to pierky 
re changes requested for arouteserver and will endeavor to delay 
resumption of the same throttling until after arouteserver users have 
had reasonable time to upgrade.

Highlights for all client developers:

  - Implement support for PeeringDB API keys:

      https://docs.peeringdb.com/howto/api_keys/

    The idea being that we will throttle users using API keys less.

  - Add a delay in between queries that is randomly between 2 and 2.5 
seconds, to reduce thundering herd. This delay will mean a client 
queries PeeringDB at most 30 hits per minute, which will be unthrottled 
   if they are authenticated using API keys and not making identical 
requests.

  - Highly preferred over separate queries: If you don't need non-public 
    contact info from PeeringDB, is that you implement peeringdb-py 
(peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side 
caching. Doing so enables you to locally query the heck out of a local 
  sqlite (or whatever) database. The start time of a peeringdb-py run 
should be randomized per the docs 
(http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use 
   peeringdb-py and here is what the once per hour update looks like for 
    all of PeeringDB:

[17/May/2022:15:40:09 +0000] "GET /api/org?since=1652794724&depth=0 
HTTP/1.1" 200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423
[17/May/2022:15:40:10 +0000] "GET /api/fac?since=1652773361&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409
[17/May/2022:15:40:10 +0000] "GET /api/net?since=1652796557&depth=0 
HTTP/1.1" 200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426
[17/May/2022:15:40:11 +0000] "GET /api/ix?since=1652397370&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397
[17/May/2022:15:40:11 +0000] "GET /api/ixfac?since=1652763759&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414
[17/May/2022:15:40:12 +0000] "GET /api/ixlan?since=1652781160&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399
[17/May/2022:15:40:12 +0000] "GET /api/ixpfx?since=1652429334&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408
[17/May/2022:15:40:13 +0000] "GET /api/netfac?since=1652790428&depth=0 
HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553
[17/May/2022:15:40:14 +0000] "GET /api/netixlan?since=1652796556&depth=0 
HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590
[17/May/2022:15:40:14 +0000] "GET /api/poc?since=1652785835&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640

    It is fast because, as I understand it, django serializes PeeringDB 
   changes, and the timestamp (since last update) results in only the 
changes being delivered.

Finally: My apology to those disrupted by this. Please feel free to 
reach out to me with any questions or concerns.

Thanks,
Chris
_______________________________________________
Pdb-tech mailing list
Pdb-tech at lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

-- 
Arnold Nipper
Chief Technology Evangelist and Co-Founder

DE-CIX Management GmbH
Lindleystraße 12 | 60314 Frankfurt a.M. | Germany
Phone +49 69 1730902 22 | Mobile +49 172 2650958
arnold.nipper at de-cix.net | www.de-cix.net
Geschaeftsfuehrer Ivaylo Ivanov und Sebastian Seifert
Registergericht AG Koeln HRB 51135

Want to work at DE-CIX: https://de-cix.net/en/about-de-cix/careers
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.peeringdb.com/pipermail/pdb-tech/attachments/20240411/eae72169/attachment.sig>