[PDB Tech] PeeringDB API throttling status and schedule

[Chris Caputo]

Based on:

  https://github.com/peeringdb/peeringdb/issues/427#issuecomment-1218382411
  
the anonymous query limit has been reverted to 20/min. Current qpm limits:

  - anonymous queries limited to 20/minute per IP address  
  - authenticated queries limited to 40/minute per user/org

Thanks,
Chris

On Mon, 15 Aug 2022, Chris Caputo wrote:
> Per the below plan, this change was just implemented:
> 
> ---
> On August 15th, adjust and watch for feedback from the community:
>   
>   - anonymous queries limited to 10/minute per IP address  
>   - authenticated queries limited to 40/minute per user/org
> ---
> 
> Please advise if you run into any issues.
> 
> Thank you,
> Chris
> 
> On Sun, 14 Aug 2022, Chris Caputo wrote:
> > Sorry - I totally jumped the gun on what UTC day it is! Reverted back to 
> > the August 8th settings:
> >   
> >   - anonymous queries limited to 20/minute per IP address  
> >   - authenticated queries limited to 60/minute per user/org
> > 
> > Chris
> > 
> > On Sun, 14 Aug 2022, Chris Caputo wrote:
> > > Per the below plan, this change was just implemented:
> > > 
> > > ---
> > > On August 15th, adjust and watch for feedback from the community:
> > > 
> > >   - anonymous queries limited to 10/minute per IP address
> > >   - authenticated queries limited to 40/minute per user/org
> > > ---
> > > 
> > > Please advise if you run into any issues.
> > > 
> > > Thank you,
> > > Chris
> > > 
> > > On Tue, 31 May 2022, Chris Caputo wrote:
> > > > After the initial introduction of PeeringDB API throttling, some software 
> > > > both open source and private, has been identified and updated. (open 
> > > > source details are below; please upgrade and encourage others to do so)
> > > > 
> > > > This API throttling is being implemented to control costs by encouraging 
> > > > efficient software design while making sure the PeeringDB resource is 
> > > > shared well. The use of API keys is being encouraged so that admins can 
> > > > reach out to users/orgs with runaway or inefficient software, and because 
> > > > it is more secure than user/pass. In addition, org API keys ease employee 
> > > > transitions.
> > > > 
> > > > Some tips for coders is below.
> > > > 
> > > > API throttling in place today:
> > > > 
> > > >   - repeated anonymous identical requests with a response size above 100k 
> > > >     are being limited to 1/hour
> > > >   - repeated anonymous identical requests of any size are being limited to 
> > > >     2/minute
> > > >   - anonymous queries are being limited to 400/minute per IP address
> > > >   - authenticated queries are being limited to 500/minute per user/org
> > > > 
> > > > Here is the current schedule of throttling changes. The schedule may 
> > > > adjust as needed as new packages that need update are discovered, so as to 
> > > > minimize disruption to the community...
> > > > 
> > > > On June 27th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 300/minute per IP address
> > > >   - authenticated queries limited to 400/minute per user/org
> > > > 
> > > > On July 11th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 200/minute per IP address
> > > >   - authenticated queries limited to 300/minute per user/org
> > > > 
> > > > On July 18th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 100/minute per IP address
> > > >   - authenticated queries limited to 200/minute per user/org
> > > > 
> > > > On July 25th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 50/minute per IP address
> > > >   - authenticated queries limited to 100/minute per user/org
> > > > 
> > > > On August 1st, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 30/minute per IP address
> > > >   - authenticated queries limited to 80/minute per user/org
> > > > 
> > > > On August 8th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 20/minute per IP address
> > > >   - authenticated queries limited to 60/minute per user/org
> > > > 
> > > > On August 15th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 10/minute per IP address
> > > >   - authenticated queries limited to 40/minute per user/org
> > > > 
> > > > Feedback/questions/concerns welcome.
> > > > 
> > > > Thanks,
> > > > Chris
> > > > 
> > > > Software:
> > > > 
> > > > - arouteserver v1.16.0: has many updates including API key support along 
> > > >   with more efficient querying.
> > > > 
> > > > - PeerFinder: API key & efficient querying patches at 
> > > >   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
> > > >   integrated.
> > > > 
> > > > Coding tips:
> > > > 
> > > > - Begin using a PeeringDB API key for all requests:
> > > > 
> > > >     https://docs.peeringdb.com/howto/api_keys/
> > > > 
> > > > - Begin performing actual caching, such as by using peeringdb-py.
> > > > 
> > > >     http://peeringdb.github.io/peeringdb-py/
> > > > 
> > > > - If unable to use a caching agent such as peeringdb-py:
> > > > 
> > > >    - Use an API key.
> > > > 
> > > >    - Set a User-Agent: header.
> > > > 
> > > >    - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
> > > >      querying 30 to 150 ASNs at a time (tune as appropriate).
> > > > 
> > > >    - Add a delay in between queries that is randomly between 2 and 2.5 
> > > >      seconds, to reduce thundering herd.
>