[PDB Tech] PeeringDB API throttling status and schedule

[Chris Caputo]

After the initial introduction of PeeringDB API throttling, some software 
both open source and private, has been identified and updated. (open 
source details are below; please upgrade and encourage others to do so)

This API throttling is being implemented to control costs by encouraging 
efficient software design while making sure the PeeringDB resource is 
shared well. The use of API keys is being encouraged so that admins can 
reach out to users/orgs with runaway or inefficient software, and because 
it is more secure than user/pass. In addition, org API keys ease employee 
transitions.

Some tips for coders is below.

API throttling in place today:

  - repeated anonymous identical requests with a response size above 100k 
    are being limited to 1/hour
  - repeated anonymous identical requests of any size are being limited to 
    2/minute
  - anonymous queries are being limited to 400/minute per IP address
  - authenticated queries are being limited to 500/minute per user/org

Here is the current schedule of throttling changes. The schedule may 
adjust as needed as new packages that need update are discovered, so as to 
minimize disruption to the community...

On June 27th, adjust and watch for feedback from the community:

  - anonymous queries limited to 300/minute per IP address
  - authenticated queries limited to 400/minute per user/org

On July 11th, adjust and watch for feedback from the community:

  - anonymous queries limited to 200/minute per IP address
  - authenticated queries limited to 300/minute per user/org

On July 18th, adjust and watch for feedback from the community:

  - anonymous queries limited to 100/minute per IP address
  - authenticated queries limited to 200/minute per user/org

On July 25th, adjust and watch for feedback from the community:

  - anonymous queries limited to 50/minute per IP address
  - authenticated queries limited to 100/minute per user/org

On August 1st, adjust and watch for feedback from the community:

  - anonymous queries limited to 30/minute per IP address
  - authenticated queries limited to 80/minute per user/org

On August 8th, adjust and watch for feedback from the community:

  - anonymous queries limited to 20/minute per IP address
  - authenticated queries limited to 60/minute per user/org

On August 15th, adjust and watch for feedback from the community:

  - anonymous queries limited to 10/minute per IP address
  - authenticated queries limited to 40/minute per user/org

Feedback/questions/concerns welcome.

Thanks,
Chris

Software:

- arouteserver v1.16.0: has many updates including API key support along 
  with more efficient querying.

- PeerFinder: API key & efficient querying patches at 
  https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
  integrated.

Coding tips:

- Begin using a PeeringDB API key for all requests:

    https://docs.peeringdb.com/howto/api_keys/

- Begin performing actual caching, such as by using peeringdb-py.

    http://peeringdb.github.io/peeringdb-py/

- If unable to use a caching agent such as peeringdb-py:

   - Use an API key.

   - Set a User-Agent: header.

   - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
     querying 30 to 150 ASNs at a time (tune as appropriate).

   - Add a delay in between queries that is randomly between 2 and 2.5 
     seconds, to reduce thundering herd.