From chriztoffer at peeringdb.com  Mon May 13 20:52:22 2024
From: chriztoffer at peeringdb.com (Chriztoffer Hansen)
Date: Mon, 13 May 2024 22:52:22 +0200
Subject: [PDB Tech] Fwd: new rate limiting mechanism is too strict
In-Reply-To: <f900a0db-4a8e-4eb7-a5a9-8b35be4a332d@de-cix.net>
References: <cd904fba-596e-9225-b7fb-d7791f8b6670@alt.net>
 <f900a0db-4a8e-4eb7-a5a9-8b35be4a332d@de-cix.net>
Message-ID: <CA+cYV6toaNaXg_5kVFLyf9_ytf9FqQ-b50M3capw+2wpP4q1Vg@mail.gmail.com>

@Arnold Nipper Is this still relevant?

The email thread is from 2022...

Chriztoffer Hansen | PeeringDB Volunteer - AdminCom Chair
chriztoffer(@)peeringdb(.)com | www.peeringdb.com

On Thu, 11 Apr 2024 at 13:28, Arnold Nipper <arnold.nipper at de-cix.net> wrote:
>
> fyi, Arnold
>
>
> -------- Weitergeleitete Nachricht --------
> Betreff: Re: [PDB Tech] new rate limiting mechanism is too strict
> Datum: Tue, 17 May 2022 16:54:59 +0000 (UTC)
> Von: Chris Caputo <ccaputo at alt.net>
> Antwort an: Chris Caputo <ccaputo-dated-1663174500.27c315 at alt.net>
> An: pdb-tech at lists.peeringdb.com
> Kopie (CC): Theo de Raadt <deraadt at yycix.ca>
>
> All,
>
> I am behind the throttling rollout in the last 24 hours, and have worked
> with Theo to loosen things up for now. I've also reached out to pierky
> re changes requested for arouteserver and will endeavor to delay
> resumption of the same throttling until after arouteserver users have
> had reasonable time to upgrade.
>
> Highlights for all client developers:
>
>   - Implement support for PeeringDB API keys:
>
>       https://docs.peeringdb.com/howto/api_keys/
>
>     The idea being that we will throttle users using API keys less.
>
>   - Add a delay in between queries that is randomly between 2 and 2.5
> seconds, to reduce thundering herd. This delay will mean a client
> queries PeeringDB at most 30 hits per minute, which will be unthrottled
>    if they are authenticated using API keys and not making identical
> requests.
>
>   - Highly preferred over separate queries: If you don't need non-public
>     contact info from PeeringDB, is that you implement peeringdb-py
> (peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side
> caching. Doing so enables you to locally query the heck out of a local
>   sqlite (or whatever) database. The start time of a peeringdb-py run
> should be randomized per the docs
> (http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use
>    peeringdb-py and here is what the once per hour update looks like for
>     all of PeeringDB:
>
> [17/May/2022:15:40:09 +0000] "GET /api/org?since=1652794724&depth=0
> HTTP/1.1" 200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423
> [17/May/2022:15:40:10 +0000] "GET /api/fac?since=1652773361&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409
> [17/May/2022:15:40:10 +0000] "GET /api/net?since=1652796557&depth=0
> HTTP/1.1" 200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426
> [17/May/2022:15:40:11 +0000] "GET /api/ix?since=1652397370&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397
> [17/May/2022:15:40:11 +0000] "GET /api/ixfac?since=1652763759&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414
> [17/May/2022:15:40:12 +0000] "GET /api/ixlan?since=1652781160&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399
> [17/May/2022:15:40:12 +0000] "GET /api/ixpfx?since=1652429334&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408
> [17/May/2022:15:40:13 +0000] "GET /api/netfac?since=1652790428&depth=0
> HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553
> [17/May/2022:15:40:14 +0000] "GET /api/netixlan?since=1652796556&depth=0
> HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590
> [17/May/2022:15:40:14 +0000] "GET /api/poc?since=1652785835&depth=0
> HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640
>
>     It is fast because, as I understand it, django serializes PeeringDB
>    changes, and the timestamp (since last update) results in only the
> changes being delivered.
>
> Finally: My apology to those disrupted by this. Please feel free to
> reach out to me with any questions or concerns.
>
> Thanks,
> Chris
> _______________________________________________
> Pdb-tech mailing list
> Pdb-tech at lists.peeringdb.com
> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
>
> --
> Arnold Nipper
> Chief Technology Evangelist and Co-Founder
>
> DE-CIX Management GmbH
> Lindleystra?e 12 | 60314 Frankfurt a.M. | Germany
> Phone +49 69 1730902 22 | Mobile +49 172 2650958
> arnold.nipper at de-cix.net | www.de-cix.net
> Geschaeftsfuehrer Ivaylo Ivanov und Sebastian Seifert
> Registergericht AG Koeln HRB 51135
>
> Want to work at DE-CIX: https://de-cix.net/en/about-de-cix/careers
> _______________________________________________
> Pdb-tech mailing list
> Pdb-tech at lists.peeringdb.com
> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

From ccaputo at alt.net  Mon May 13 23:53:26 2024
From: ccaputo at alt.net (Chris Caputo)
Date: Mon, 13 May 2024 23:53:26 +0000 (UTC)
Subject: [PDB Tech] Fwd: new rate limiting mechanism is too strict
In-Reply-To: <CA+cYV6toaNaXg_5kVFLyf9_ytf9FqQ-b50M3capw+2wpP4q1Vg@mail.gmail.com>
References: <cd904fba-596e-9225-b7fb-d7791f8b6670@alt.net>
 <f900a0db-4a8e-4eb7-a5a9-8b35be4a332d@de-cix.net>
 <CA+cYV6toaNaXg_5kVFLyf9_ytf9FqQ-b50M3capw+2wpP4q1Vg@mail.gmail.com>
Message-ID: <a14d2f8f-d85-8a11-41b1-e0b4a1b34a1c@alt.net>

I'd say it is.

Chris

On Mon, 13 May 2024, Chriztoffer Hansen wrote:
> @Arnold Nipper Is this still relevant?
> 
> The email thread is from 2022...
> 
> Chriztoffer Hansen | PeeringDB Volunteer - AdminCom Chair
> chriztoffer(@)peeringdb(.)com | www.peeringdb.com
> 
> On Thu, 11 Apr 2024 at 13:28, Arnold Nipper <arnold.nipper at de-cix.net> wrote:
> >
> > fyi, Arnold
> >
> >
> > -------- Weitergeleitete Nachricht --------
> > Betreff: Re: [PDB Tech] new rate limiting mechanism is too strict
> > Datum: Tue, 17 May 2022 16:54:59 +0000 (UTC)
> > Von: Chris Caputo <ccaputo at alt.net>
> > Antwort an: Chris Caputo <ccaputo-dated-1663174500.27c315 at alt.net>
> > An: pdb-tech at lists.peeringdb.com
> > Kopie (CC): Theo de Raadt <deraadt at yycix.ca>
> >
> > All,
> >
> > I am behind the throttling rollout in the last 24 hours, and have worked
> > with Theo to loosen things up for now. I've also reached out to pierky
> > re changes requested for arouteserver and will endeavor to delay
> > resumption of the same throttling until after arouteserver users have
> > had reasonable time to upgrade.
> >
> > Highlights for all client developers:
> >
> >   - Implement support for PeeringDB API keys:
> >
> >       https://docs.peeringdb.com/howto/api_keys/
> >
> >     The idea being that we will throttle users using API keys less.
> >
> >   - Add a delay in between queries that is randomly between 2 and 2.5
> > seconds, to reduce thundering herd. This delay will mean a client
> > queries PeeringDB at most 30 hits per minute, which will be unthrottled
> >    if they are authenticated using API keys and not making identical
> > requests.
> >
> >   - Highly preferred over separate queries: If you don't need non-public
> >     contact info from PeeringDB, is that you implement peeringdb-py
> > (peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side
> > caching. Doing so enables you to locally query the heck out of a local
> >   sqlite (or whatever) database. The start time of a peeringdb-py run
> > should be randomized per the docs
> > (http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use
> >    peeringdb-py and here is what the once per hour update looks like for
> >     all of PeeringDB:
> >
> > [17/May/2022:15:40:09 +0000] "GET /api/org?since=1652794724&depth=0
> > HTTP/1.1" 200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423
> > [17/May/2022:15:40:10 +0000] "GET /api/fac?since=1652773361&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409
> > [17/May/2022:15:40:10 +0000] "GET /api/net?since=1652796557&depth=0
> > HTTP/1.1" 200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426
> > [17/May/2022:15:40:11 +0000] "GET /api/ix?since=1652397370&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397
> > [17/May/2022:15:40:11 +0000] "GET /api/ixfac?since=1652763759&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414
> > [17/May/2022:15:40:12 +0000] "GET /api/ixlan?since=1652781160&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399
> > [17/May/2022:15:40:12 +0000] "GET /api/ixpfx?since=1652429334&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408
> > [17/May/2022:15:40:13 +0000] "GET /api/netfac?since=1652790428&depth=0
> > HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553
> > [17/May/2022:15:40:14 +0000] "GET /api/netixlan?since=1652796556&depth=0
> > HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590
> > [17/May/2022:15:40:14 +0000] "GET /api/poc?since=1652785835&depth=0
> > HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640
> >
> >     It is fast because, as I understand it, django serializes PeeringDB
> >    changes, and the timestamp (since last update) results in only the
> > changes being delivered.
> >
> > Finally: My apology to those disrupted by this. Please feel free to
> > reach out to me with any questions or concerns.
> >
> > Thanks,
> > Chris
> > _______________________________________________
> > Pdb-tech mailing list
> > Pdb-tech at lists.peeringdb.com
> > https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> >
> > --
> > Arnold Nipper
> > Chief Technology Evangelist and Co-Founder
> >
> > DE-CIX Management GmbH
> > Lindleystra?e 12 | 60314 Frankfurt a.M. | Germany
> > Phone +49 69 1730902 22 | Mobile +49 172 2650958
> > arnold.nipper at de-cix.net | www.de-cix.net
> > Geschaeftsfuehrer Ivaylo Ivanov und Sebastian Seifert
> > Registergericht AG Koeln HRB 51135
> >
> > Want to work at DE-CIX: https://de-cix.net/en/about-de-cix/careers
> > _______________________________________________
> > Pdb-tech mailing list
> > Pdb-tech at lists.peeringdb.com
> > https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> _______________________________________________
> Pdb-tech mailing list
> Pdb-tech at lists.peeringdb.com
> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
>