[PDB Tech] Fwd: new rate limiting mechanism is too strict

Mon May 13 20:52:22 UTC 2024

@Arnold Nipper Is this still relevant?

The email thread is from 2022...

Chriztoffer Hansen | PeeringDB Volunteer - AdminCom Chair
chriztoffer(@)peeringdb(.)com | www.peeringdb.com

On Thu, 11 Apr 2024 at 13:28, Arnold Nipper <arnold.nipper at de-cix.net> wrote:
>
> fyi, Arnold
>
>
> -------- Weitergeleitete Nachricht --------
> Betreff: Re: [PDB Tech] new rate limiting mechanism is too strict
> Datum: Tue, 17 May 2022 16:54:59 +0000 (UTC)
> Von: Chris Caputo <ccaputo at alt.net>
> Antwort an: Chris Caputo <ccaputo-dated-1663174500.27c315 at alt.net>
> An: pdb-tech at lists.peeringdb.com
> Kopie (CC): Theo de Raadt <deraadt at yycix.ca>
>
> All,
>
> I am behind the throttling rollout in the last 24 hours, and have worked
> with Theo to loosen things up for now. I've also reached out to pierky
> re changes requested for arouteserver and will endeavor to delay
> resumption of the same throttling until after arouteserver users have
> had reasonable time to upgrade.
>
> Highlights for all client developers:
>
>   - Implement support for PeeringDB API keys:
>
>       https://docs.peeringdb.com/howto/api_keys/
>
>     The idea being that we will throttle users using API keys less.
>
>   - Add a delay in between queries that is randomly between 2 and 2.5
> seconds, to reduce thundering herd. This delay will mean a client
> queries PeeringDB at most 30 hits per minute, which will be unthrottled
>    if they are authenticated using API keys and not making identical
> requests.
>
>   - Highly preferred over separate queries: If you don't need non-public
>     contact info from PeeringDB, is that you implement peeringdb-py
> (peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side
> caching. Doing so enables you to locally query the heck out of a local
>   sqlite (or whatever) database. The start time of a peeringdb-py run
> should be randomized per the docs
> (http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use
>    peeringdb-py and here is what the once per hour update looks like for
>     all of PeeringDB:
>
> [17/May/2022:15:40:09 +0000] "GET /api/org?since=1652794724&depth=0
> HTTP/1.1" 200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423
> [17/May/2022:15:40:10 +0000] "GET /api/fac?since=1652773361&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409
> [17/May/2022:15:40:10 +0000] "GET /api/net?since=1652796557&depth=0
> HTTP/1.1" 200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426
> [17/May/2022:15:40:11 +0000] "GET /api/ix?since=1652397370&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397
> [17/May/2022:15:40:11 +0000] "GET /api/ixfac?since=1652763759&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414
> [17/May/2022:15:40:12 +0000] "GET /api/ixlan?since=1652781160&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399
> [17/May/2022:15:40:12 +0000] "GET /api/ixpfx?since=1652429334&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408
> [17/May/2022:15:40:13 +0000] "GET /api/netfac?since=1652790428&depth=0
> HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553
> [17/May/2022:15:40:14 +0000] "GET /api/netixlan?since=1652796556&depth=0
> HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590
> [17/May/2022:15:40:14 +0000] "GET /api/poc?since=1652785835&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640
>
>     It is fast because, as I understand it, django serializes PeeringDB
>    changes, and the timestamp (since last update) results in only the
> changes being delivered.
>
> Finally: My apology to those disrupted by this. Please feel free to
> reach out to me with any questions or concerns.
>
> Thanks,
> Chris
> _______________________________________________
> Pdb-tech mailing list
> Pdb-tech at lists.peeringdb.com
> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
>
> --
> Arnold Nipper
> Chief Technology Evangelist and Co-Founder
>
> DE-CIX Management GmbH
> Lindleystraße 12 | 60314 Frankfurt a.M. | Germany
> Phone +49 69 1730902 22 | Mobile +49 172 2650958
> arnold.nipper at de-cix.net | www.de-cix.net
> Geschaeftsfuehrer Ivaylo Ivanov und Sebastian Seifert
> Registergericht AG Koeln HRB 51135
>
> Want to work at DE-CIX: https://de-cix.net/en/about-de-cix/careers
> _______________________________________________
> Pdb-tech mailing list
> Pdb-tech at lists.peeringdb.com
> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech