August 2022 - Pdb-tech

new rate limiting mechanism is too strict
by Theo de Raadt 13 May '24

13 May '24

At YYCIX, we run arouteserver, which polls peeringdb. On May 15 (I think) we started seeing failures to download peeringdb records, which has resulted in our routeserver configuration not being updated as usual. I've become aware that the API has a new rate filtering mechanism, for non-APIKEY accesses. Well, arouteserver doesn't do APIKEY, and I don't see how the author of arouteserver would have received any notice that this was suddenly mandatory, nor all the arouteserver users. I think peeringdb should have looked more carefully into the no-APIKEY accesses to determine what downstream effects might occur from this change. Testing further manually, I observe some really crazy behaviour. Using a regular browser, I can keep reloading the same record over and over without hitting any limit. Using a command-line client (curl or OpenBSD ftp(1)), i hit some extremely strict limits very quickly. On the 11th lookup, I receiving this, and it started counting down: {"message": "Request was throttled. Expected available in 59 minutes. Authenticate for less restrictions. For more information: https://docs.peeringdb.com/howto/api_keys/", "meta": {"error": "Too Many Requests"}} All requests are blocked for an hour. I paused, and tried a new lookup every 5 minutes. Half an hour later it allowed me to retrieve 1 record, but then continued the countdown: {"message": "Request was throttled. Expected available in 17 minutes. Authenticate for less restrictions. For more information: https://docs.peeringdb.com/howto/api_keys/", "meta": {"error": "Too Many Requests"}} If the new limit is 10 records an hour, I mean you could really just shut down the service entirely, there's really no difference. I understand using APIKEY is the new way, but the old way has effectively been disabled without notice. But also, why does this work in the browser? I can reload hundreds of times. Is the official browser User-agent being considered exempt from the rule? If so I don't understand what the protection plan is.. maybe we should all change our commandline tools to declare they are Chrome?

5 6

Re: [PDB Tech] PeeringDB API throttling status and schedule
by Chris Caputo 18 Aug '22

18 Aug '22

Per the below plan, this change was just implemented: --- On August 15th, adjust and watch for feedback from the community: - anonymous queries limited to 10/minute per IP address - authenticated queries limited to 40/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > > - Use an API key. > > - Set a User-Agent: header. > > - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > > - Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd.

1 3

non-canonical queries losing auth on redirect to canonical
by Chris Caputo 09 Aug '22

09 Aug '22

It is important to query against the canonical "www.peeringdb.com". Python and presumably other libraries will drop the authentication request header on a redirect, as a safety precaution: https://github.com/psf/requests/blob/177dd90f/requests/sessions.py#L284-L296 Unfortunately, when this happens is opaque because PeeringDB will try to satisfy what it thinks is simply an anonymous request. If you have an opinion on whether it makes sense for PeeringDB to stop redirecting API requests, check out: https://github.com/peeringdb/peeringdb/issues/1139 Chris

1 0

Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)
by Chris Caputo 09 Aug '22

09 Aug '22

Per the below plan, this change was just implemented: --- On August 8th, adjust and watch for feedback from the community: - anonymous queries limited to 20/minute per IP address - authenticated queries limited to 60/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > > - Use an API key. > > - Set a User-Agent: header. > > - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > > - Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd.

3 4

Re: [PDB Tech] PeeringDB API throttling status and schedule
by Chris Caputo 08 Aug '22

08 Aug '22

[resend] Per the below plan, this change was just implemented: --- On August 8th, adjust and watch for feedback from the community: - anonymous queries limited to 20/minute per IP address - authenticated queries limited to 60/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > > - Use an API key. > > - Set a User-Agent: header. > > - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > > - Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd.

1 1

Re: [PDB Tech] PeeringDB API throttling status and schedule
by Chris Caputo 01 Aug '22

01 Aug '22

Per the below plan, this change was just implemented: --- On August 1st, adjust and watch for feedback from the community: - anonymous queries limited to 30/minute per IP address - authenticated queries limited to 80/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > > - Use an API key. > > - Set a User-Agent: header. > > - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > > - Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd.

1 0