[PDB Tech] PeeringDB API throttling status and schedule (fwd)
Dale W. Carder
dwcarder at es.net
Tue Aug 9 12:37:08 PDT 2022
Thus spake Chris Caputo (ccaputo at alt.net) on Tue, Aug 09, 2022 at 07:11:02PM +0000:
> Dale, if you are getting 200 for an obviously bad api-key, then the
> authentication format is not correct. Examples/details at:
>
> https://github.com/peeringdb/peeringdb/issues/1220#issuecomment-1209763911
>
> With a correctly formated auth request, 401 (unauth) will be returned for
> a bad key.
>
> Please reach out to me privately with your source IP if you'd like me to
> review how the server sees your requests, or for efficiency efforts, or if
> you need any help getting api-key authentication working.
Will do! I really appreciate it. Like many things, hopefully it's
it's mostly PEBCAK.
Dale
> On Tue, 9 Aug 2022, Stephen McManus wrote:
> > > However, for a read-only API key, how does one know if it's working?
> > > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I
> > > got results back vs a 4xx error code. So from an error handling
> > > perspective it seems hard to gauge if I am using a valid api key
> > > getting premium service vs an invalid api key quietly lumped into
> > > the anonymous rate-limit bucket.
> >
> > This is something we should fix. I've filed https://github.com/peeringdb/peeringdb/issues/1220 to get it addressed
> >
> > -Steve
> >
> >
> >
> >
> > > On Aug 9, 2022, at 1:56 PM, Dale W. Carder <dwcarder at es.net> wrote:
> > >
> > > Thus spake Chris Caputo (ccaputo at alt.net) on Mon, Aug 08, 2022 at 04:41:17PM +0000:
> > >> Per the below plan, this change was just implemented:
> > >>
> > >> ---
> > >> On August 8th, adjust and watch for feedback from the community:
> > >>
> > >> - anonymous queries limited to 20/minute per IP address
> > >> - authenticated queries limited to 60/minute per user/org
> > >> ---
> > >>
> > >> Please advise if you run into any issues.
> > >
> > > This is about where I start to get concerned. First off, I'm not
> > > sure how well communicated this was. I'd like to think that I'm
> > > generally aware of what's happening in our ecosystem, but someone
> > > (thankfully) had to point this out to me.
> > >
> > > So, our provisioning code is perhaps naive... jobs are dispatched
> > > into a task queue where they are run to completion, one per ASN.
> > > At present it would be non-trivial to implement a bulk query to
> > > cache ahead of time (making peeringdb lookups asynchronous), but
> > > that absolutely is on our longer-term roadmap. It's also not the
> > > easiest to rate-limit the queue as only some of them actually need
> > > a peeringdb lookup (a huge amount of our peers are private asn
> > > and/or in a non-dfz l3vpn's), but we have limited the concurrency
> > > and can count on the general case that our code is reassuringly
> > > slow.
> > >
> > > Luckily, some of the other things suggested below are easy, and I
> > > was testing it out today. We'll set a custom user-agent, limit
> > > our query to only the fields we care about, and use an api key.
> > >
> > > However, for a read-only API key, how does one know if it's working?
> > > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I
> > > got results back vs a 4xx error code. So from an error handling
> > > perspective it seems hard to gauge if I am using a valid api key
> > > getting premium service vs an invalid api key quietly lumped into
> > > the anonymous rate-limit bucket.
> > >
> > > Dale
> > >
> > >
> > >> On Tue, 31 May 2022, Chris Caputo wrote:
> > >>> After the initial introduction of PeeringDB API throttling, some software
> > >>> both open source and private, has been identified and updated. (open
> > >>> source details are below; please upgrade and encourage others to do so)
> > >>>
> > >>> This API throttling is being implemented to control costs by encouraging
> > >>> efficient software design while making sure the PeeringDB resource is
> > >>> shared well. The use of API keys is being encouraged so that admins can
> > >>> reach out to users/orgs with runaway or inefficient software, and because
> > >>> it is more secure than user/pass. In addition, org API keys ease employee
> > >>> transitions.
> > >>>
> > >>> Some tips for coders is below.
> > >>>
> > >>> API throttling in place today:
> > >>>
> > >>> - repeated anonymous identical requests with a response size above 100k
> > >>> are being limited to 1/hour
> > >>> - repeated anonymous identical requests of any size are being limited to
> > >>> 2/minute
> > >>> - anonymous queries are being limited to 400/minute per IP address
> > >>> - authenticated queries are being limited to 500/minute per user/org
> > >>>
> > >>> Here is the current schedule of throttling changes. The schedule may
> > >>> adjust as needed as new packages that need update are discovered, so as to
> > >>> minimize disruption to the community...
> > >>>
> > >>> On June 27th, adjust and watch for feedback from the community:
> > >>>
> > >>> - anonymous queries limited to 300/minute per IP address
> > >>> - authenticated queries limited to 400/minute per user/org
> > >>>
> > >>> On July 11th, adjust and watch for feedback from the community:
> > >>>
> > >>> - anonymous queries limited to 200/minute per IP address
> > >>> - authenticated queries limited to 300/minute per user/org
> > >>>
> > >>> On July 18th, adjust and watch for feedback from the community:
> > >>>
> > >>> - anonymous queries limited to 100/minute per IP address
> > >>> - authenticated queries limited to 200/minute per user/org
> > >>>
> > >>> On July 25th, adjust and watch for feedback from the community:
> > >>>
> > >>> - anonymous queries limited to 50/minute per IP address
> > >>> - authenticated queries limited to 100/minute per user/org
> > >>>
> > >>> On August 1st, adjust and watch for feedback from the community:
> > >>>
> > >>> - anonymous queries limited to 30/minute per IP address
> > >>> - authenticated queries limited to 80/minute per user/org
> > >>>
> > >>> On August 8th, adjust and watch for feedback from the community:
> > >>>
> > >>> - anonymous queries limited to 20/minute per IP address
> > >>> - authenticated queries limited to 60/minute per user/org
> > >>>
> > >>> On August 15th, adjust and watch for feedback from the community:
> > >>>
> > >>> - anonymous queries limited to 10/minute per IP address
> > >>> - authenticated queries limited to 40/minute per user/org
> > >>>
> > >>> Feedback/questions/concerns welcome.
> > >>>
> > >>> Thanks,
> > >>> Chris
> > >>>
> > >>> Software:
> > >>>
> > >>> - arouteserver v1.16.0: has many updates including API key support along
> > >>> with more efficient querying.
> > >>>
> > >>> - PeerFinder: API key & efficient querying patches at
> > >>> https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be
> > >>> integrated.
> > >>>
> > >>> Coding tips:
> > >>>
> > >>> - Begin using a PeeringDB API key for all requests:
> > >>>
> > >>> https://docs.peeringdb.com/howto/api_keys/
> > >>>
> > >>> - Begin performing actual caching, such as by using peeringdb-py.
> > >>>
> > >>> http://peeringdb.github.io/peeringdb-py/
> > >>>
> > >>> - If unable to use a caching agent such as peeringdb-py:
> > >>>
> > >>> - Use an API key.
> > >>>
> > >>> - Set a User-Agent: header.
> > >>>
> > >>> - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by
> > >>> querying 30 to 150 ASNs at a time (tune as appropriate).
> > >>>
> > >>> - Add a delay in between queries that is randomly between 2 and 2.5
> > >>> seconds, to reduce thundering herd.
> > >> _______________________________________________
> > >> Pdb-tech mailing list
> > >> Pdb-tech at lists.peeringdb.com
> > >> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> > > _______________________________________________
> > > Pdb-tech mailing list
> > > Pdb-tech at lists.peeringdb.com
> > > https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> >
> > _______________________________________________
> > Pdb-tech mailing list
> > Pdb-tech at lists.peeringdb.com
> > https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> >
More information about the Pdb-tech
mailing list