[PDB Tech] Fwd: new rate limiting mechanism is too strict

Mon May 13 23:53:26 UTC 2024

I'd say it is.

Chris

On Mon, 13 May 2024, Chriztoffer Hansen wrote:
> @Arnold Nipper Is this still relevant?
> 
> The email thread is from 2022...
> 
> Chriztoffer Hansen | PeeringDB Volunteer - AdminCom Chair
> chriztoffer(@)peeringdb(.)com | www.peeringdb.com
> 
> On Thu, 11 Apr 2024 at 13:28, Arnold Nipper <arnold.nipper at de-cix.net> wrote:
> >
> > fyi, Arnold
> >
> >
> > -------- Weitergeleitete Nachricht --------
> > Betreff: Re: [PDB Tech] new rate limiting mechanism is too strict
> > Datum: Tue, 17 May 2022 16:54:59 +0000 (UTC)
> > Von: Chris Caputo <ccaputo at alt.net>
> > Antwort an: Chris Caputo <ccaputo-dated-1663174500.27c315 at alt.net>
> > An: pdb-tech at lists.peeringdb.com
> > Kopie (CC): Theo de Raadt <deraadt at yycix.ca>
> >
> > All,
> >
> > I am behind the throttling rollout in the last 24 hours, and have worked
> > with Theo to loosen things up for now. I've also reached out to pierky
> > re changes requested for arouteserver and will endeavor to delay
> > resumption of the same throttling until after arouteserver users have
> > had reasonable time to upgrade.
> >
> > Highlights for all client developers:
> >
> >   - Implement support for PeeringDB API keys:
> >
> >       https://docs.peeringdb.com/howto/api_keys/
> >
> >     The idea being that we will throttle users using API keys less.
> >
> >   - Add a delay in between queries that is randomly between 2 and 2.5
> > seconds, to reduce thundering herd. This delay will mean a client
> > queries PeeringDB at most 30 hits per minute, which will be unthrottled
> >    if they are authenticated using API keys and not making identical
> > requests.
> >
> >   - Highly preferred over separate queries: If you don't need non-public
> >     contact info from PeeringDB, is that you implement peeringdb-py
> > (peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side
> > caching. Doing so enables you to locally query the heck out of a local
> >   sqlite (or whatever) database. The start time of a peeringdb-py run
> > should be randomized per the docs
> > (http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use
> >    peeringdb-py and here is what the once per hour update looks like for
> >     all of PeeringDB:
> >
> > [17/May/2022:15:40:09 +0000] "GET /api/org?since=1652794724&depth=0
> > HTTP/1.1" 200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423
> > [17/May/2022:15:40:10 +0000] "GET /api/fac?since=1652773361&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409
> > [17/May/2022:15:40:10 +0000] "GET /api/net?since=1652796557&depth=0
> > HTTP/1.1" 200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426
> > [17/May/2022:15:40:11 +0000] "GET /api/ix?since=1652397370&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397
> > [17/May/2022:15:40:11 +0000] "GET /api/ixfac?since=1652763759&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414
> > [17/May/2022:15:40:12 +0000] "GET /api/ixlan?since=1652781160&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399
> > [17/May/2022:15:40:12 +0000] "GET /api/ixpfx?since=1652429334&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408
> > [17/May/2022:15:40:13 +0000] "GET /api/netfac?since=1652790428&depth=0
> > HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553
> > [17/May/2022:15:40:14 +0000] "GET /api/netixlan?since=1652796556&depth=0
> > HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590
> > [17/May/2022:15:40:14 +0000] "GET /api/poc?since=1652785835&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640
> >
> >     It is fast because, as I understand it, django serializes PeeringDB
> >    changes, and the timestamp (since last update) results in only the
> > changes being delivered.
> >
> > Finally: My apology to those disrupted by this. Please feel free to
> > reach out to me with any questions or concerns.
> >
> > Thanks,
> > Chris
> > _______________________________________________
> > Pdb-tech mailing list
> > Pdb-tech at lists.peeringdb.com
> > https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> >
> > --
> > Arnold Nipper
> > Chief Technology Evangelist and Co-Founder
> >
> > DE-CIX Management GmbH
> > Lindleystraße 12 | 60314 Frankfurt a.M. | Germany
> > Phone +49 69 1730902 22 | Mobile +49 172 2650958
> > arnold.nipper at de-cix.net | www.de-cix.net
> > Geschaeftsfuehrer Ivaylo Ivanov und Sebastian Seifert
> > Registergericht AG Koeln HRB 51135
> >
> > Want to work at DE-CIX: https://de-cix.net/en/about-de-cix/careers
> > _______________________________________________
> > Pdb-tech mailing list
> > Pdb-tech at lists.peeringdb.com
> > https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> _______________________________________________
> Pdb-tech mailing list
> Pdb-tech at lists.peeringdb.com
> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> 