From tstrickx at cloudflare.com  Thu Apr 11 11:21:49 2024
From: tstrickx at cloudflare.com (Tom Strickx)
Date: Thu, 11 Apr 2024 12:21:49 +0100
Subject: [PDB Tech] Question about API ratelimits
Message-ID: <CAC93g0QUyskHdruLgB-M1rXHZstPtZMDPdt2sQ9OrH8DpBQfoQ@mail.gmail.com>

Hey folks,

We're wondering how the ratelimits are enforced these days.
Specifically, authenticated (so with API key) requests. Are these enforced
by API-key, by AccountID, by IP, by subnet, by star sign, ...?
Let me know if there's some piece of documentation documenting all of this.

Thanks!
-- 
Tom Strickx
Principal Network Engineer
AS13335 - Cloudflare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.peeringdb.com/pipermail/pdb-tech/attachments/20240411/4f41eac1/attachment.htm>

From arnold.nipper at de-cix.net  Thu Apr 11 11:28:12 2024
From: arnold.nipper at de-cix.net (Arnold Nipper)
Date: Thu, 11 Apr 2024 13:28:12 +0200
Subject: [PDB Tech] Fwd:  new rate limiting mechanism is too strict
In-Reply-To: <cd904fba-596e-9225-b7fb-d7791f8b6670@alt.net>
References: <cd904fba-596e-9225-b7fb-d7791f8b6670@alt.net>
Message-ID: <f900a0db-4a8e-4eb7-a5a9-8b35be4a332d@de-cix.net>

fyi, Arnold


-------- Weitergeleitete Nachricht --------
Betreff: Re: [PDB Tech] new rate limiting mechanism is too strict
Datum: Tue, 17 May 2022 16:54:59 +0000 (UTC)
Von: Chris Caputo <ccaputo at alt.net>
Antwort an: Chris Caputo <ccaputo-dated-1663174500.27c315 at alt.net>
An: pdb-tech at lists.peeringdb.com
Kopie (CC): Theo de Raadt <deraadt at yycix.ca>

All,

I am behind the throttling rollout in the last 24 hours, and have worked 
with Theo to loosen things up for now. I've also reached out to pierky 
re changes requested for arouteserver and will endeavor to delay 
resumption of the same throttling until after arouteserver users have 
had reasonable time to upgrade.

Highlights for all client developers:

  - Implement support for PeeringDB API keys:

      https://docs.peeringdb.com/howto/api_keys/

    The idea being that we will throttle users using API keys less.

  - Add a delay in between queries that is randomly between 2 and 2.5 
seconds, to reduce thundering herd. This delay will mean a client 
queries PeeringDB at most 30 hits per minute, which will be unthrottled 
   if they are authenticated using API keys and not making identical 
requests.

  - Highly preferred over separate queries: If you don't need non-public 
    contact info from PeeringDB, is that you implement peeringdb-py 
(peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side 
caching. Doing so enables you to locally query the heck out of a local 
  sqlite (or whatever) database. The start time of a peeringdb-py run 
should be randomized per the docs 
(http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use 
   peeringdb-py and here is what the once per hour update looks like for 
    all of PeeringDB:

[17/May/2022:15:40:09 +0000] "GET /api/org?since=1652794724&depth=0 
HTTP/1.1" 200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423
[17/May/2022:15:40:10 +0000] "GET /api/fac?since=1652773361&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409
[17/May/2022:15:40:10 +0000] "GET /api/net?since=1652796557&depth=0 
HTTP/1.1" 200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426
[17/May/2022:15:40:11 +0000] "GET /api/ix?since=1652397370&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397
[17/May/2022:15:40:11 +0000] "GET /api/ixfac?since=1652763759&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414
[17/May/2022:15:40:12 +0000] "GET /api/ixlan?since=1652781160&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399
[17/May/2022:15:40:12 +0000] "GET /api/ixpfx?since=1652429334&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408
[17/May/2022:15:40:13 +0000] "GET /api/netfac?since=1652790428&depth=0 
HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553
[17/May/2022:15:40:14 +0000] "GET /api/netixlan?since=1652796556&depth=0 
HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590
[17/May/2022:15:40:14 +0000] "GET /api/poc?since=1652785835&depth=0 
HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640

    It is fast because, as I understand it, django serializes PeeringDB 
   changes, and the timestamp (since last update) results in only the 
changes being delivered.

Finally: My apology to those disrupted by this. Please feel free to 
reach out to me with any questions or concerns.

Thanks,
Chris
_______________________________________________
Pdb-tech mailing list
Pdb-tech at lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

-- 
Arnold Nipper
Chief Technology Evangelist and Co-Founder

DE-CIX Management GmbH
Lindleystra?e 12 | 60314 Frankfurt a.M. | Germany
Phone +49 69 1730902 22 | Mobile +49 172 2650958
arnold.nipper at de-cix.net | www.de-cix.net
Geschaeftsfuehrer Ivaylo Ivanov und Sebastian Seifert
Registergericht AG Koeln HRB 51135

Want to work at DE-CIX: https://de-cix.net/en/about-de-cix/careers
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <http://lists.peeringdb.com/pipermail/pdb-tech/attachments/20240411/eae72169/attachment.sig>

From ccaputo at alt.net  Thu Apr 11 14:11:25 2024
From: ccaputo at alt.net (Chris Caputo)
Date: Thu, 11 Apr 2024 14:11:25 +0000 (UTC)
Subject: [PDB Tech] Question about API ratelimits
In-Reply-To: <CAC93g0QUyskHdruLgB-M1rXHZstPtZMDPdt2sQ9OrH8DpBQfoQ@mail.gmail.com>
References: <CAC93g0QUyskHdruLgB-M1rXHZstPtZMDPdt2sQ9OrH8DpBQfoQ@mail.gmail.com>
Message-ID: <5aa9704e-2e14-8932-22fc-2bee593a7376@alt.net>

On Thu, 11 Apr 2024, Tom Strickx wrote:
> Hey folks,
> 
> We're wondering how the ratelimits are enforced these days.
> Specifically, authenticated (so with API key) requests. Are these enforced
> by API-key, by AccountID, by IP, by subnet, by star sign, ...?
> Let me know if there's some piece of documentation documenting all of this.
> 
> Thanks!
> -- 
> Tom Strickx
> Principal Network Engineer
> AS13335 - Cloudflare

Hi Tom,

Per:

  https://docs.peeringdb.com/howto/work_within_peeringdbs_query_limits/

  - Anonymous queries limited to 20/minute per IP address

  - Authenticated queries limited to 40/minute per user or organization 
    (when an organizational API key is used)

This comes from a set of HOWTOs that may be of interest to others, 
including one on the caching software peeringdb-py:

  https://docs.peeringdb.com/howtos/

The Seattle IX uses peeringdb-py to perform many queries of PeeringDB per 
day to inform its web site and route servers, with nil impact to PeeringDB 
itself, since the queries hit a local database instead.

There are also query limits for repeated identical from unauthenticated 
queries. These are per IP address and per /24 or /64 address block.

If you want to dig into the Django code for throttling, check out:

  https://github.com/peeringdb/peeringdb/blob/master/peeringdb_server/rest_throttles.py

Let me know if you have other questions. I volunteer on PDB Ops.

Chris