[PDB Tech] PeeringDB API throttling status and schedule

[Jörg Kost]

Hi Chris,

Is there a basis for calculating why there should only be 40 requests 
for authorized participants at the end? Also, is the Query_String 
limited to some maximum size?

When I benchmark it, even with the maximum utilization of 150 ASN 
numbers in the query list for a large IX like DE-CIX, I see about ten 
queries with ASN_LIST, including the IX and NetIX queries. With that, we 
would have already exhausted 25% of the volume.

My general suggestion would be that we leave a bit more headroom for 
requests in the same period without a self-throttling penalty. The 
target value should conclude at 10% of the queries for the largest IX as 
a variable;  therefore, in 2022, at least 100 ~ 120 requests per minute 
shall be allowed.

I wrote https://github.com/ipcjk/ixgen half a decade ago (god, how time 
flies). I patched in the API keys yesterday; ASN_LIST will also be 
included in the next release. However, there is another significant 
advantage; the thing works with a local cache of the JSON files from 
PeeringDB. It can be used as a simple API server directly as a binary 
with compatible queries. So you can quickly get rid of 1000+ queries in 
a few seconds without SQL, other dependencies, and bugging the original 
peeringDB-source.

BR Jörg

On 31 May 2022, at 21:31, Chris Caputo wrote:

> After the initial introduction of PeeringDB API throttling, some 
> software
> both open source and private, has been identified and updated. (open
> source details are below; please upgrade and encourage others to do 
> so)
>
> This API throttling is being implemented to control costs by 
> encouraging
> efficient software design while making sure the PeeringDB resource is
> shared well. The use of API keys is being encouraged so that admins 
> can
> reach out to users/orgs with runaway or inefficient software, and 
> because
> it is more secure than user/pass. In addition, org API keys ease 
> employee
> transitions.
>
> Some tips for coders is below.
>
> API throttling in place today:
>
>   - repeated anonymous identical requests with a response size above 
> 100k
>     are being limited to 1/hour
>   - repeated anonymous identical requests of any size are being 
> limited to
>     2/minute
>   - anonymous queries are being limited to 400/minute per IP address
>   - authenticated queries are being limited to 500/minute per user/org
>
> Here is the current schedule of throttling changes. The schedule may
> adjust as needed as new packages that need update are discovered, so 
> as to
> minimize disruption to the community...
>
> On June 27th, adjust and watch for feedback from the community:
>
>   - anonymous queries limited to 300/minute per IP address
>   - authenticated queries limited to 400/minute per user/org
>
> On July 11th, adjust and watch for feedback from the community:
>
>   - anonymous queries limited to 200/minute per IP address
>   - authenticated queries limited to 300/minute per user/org
>
> On July 18th, adjust and watch for feedback from the community:
>
>   - anonymous queries limited to 100/minute per IP address
>   - authenticated queries limited to 200/minute per user/org
>
> On July 25th, adjust and watch for feedback from the community:
>
>   - anonymous queries limited to 50/minute per IP address
>   - authenticated queries limited to 100/minute per user/org
>
> On August 1st, adjust and watch for feedback from the community:
>
>   - anonymous queries limited to 30/minute per IP address
>   - authenticated queries limited to 80/minute per user/org
>
> On August 8th, adjust and watch for feedback from the community:
>
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
>
> On August 15th, adjust and watch for feedback from the community:
>
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
>
> Feedback/questions/concerns welcome.
>
> Thanks,
> Chris
>
> Software:
>
> - arouteserver v1.16.0: has many updates including API key support 
> along
>   with more efficient querying.
>
> - PeerFinder: API key & efficient querying patches at
>   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be
>   integrated.
>
> Coding tips:
>
> - Begin using a PeeringDB API key for all requests:
>
>     https://docs.peeringdb.com/howto/api_keys/
>
> - Begin performing actual caching, such as by using peeringdb-py.
>
>     http://peeringdb.github.io/peeringdb-py/
>
> - If unable to use a caching agent such as peeringdb-py:
>
>    - Use an API key.
>
>    - Set a User-Agent: header.
>
>    - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by
>      querying 30 to 150 ASNs at a time (tune as appropriate).
>
>    - Add a delay in between queries that is randomly between 2 and 2.5
>      seconds, to reduce thundering herd.
> _______________________________________________
> Pdb-tech mailing list
> Pdb-tech at lists.peeringdb.com
> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech