September 2022 - Pdb-tech

new rate limiting mechanism is too strict
by Theo de Raadt 13 May '24

13 May '24

At YYCIX, we run arouteserver, which polls peeringdb. On May 15 (I think) we started seeing failures to download peeringdb records, which has resulted in our routeserver configuration not being updated as usual. I've become aware that the API has a new rate filtering mechanism, for non-APIKEY accesses. Well, arouteserver doesn't do APIKEY, and I don't see how the author of arouteserver would have received any notice that this was suddenly mandatory, nor all the arouteserver users. I think peeringdb should have looked more carefully into the no-APIKEY accesses to determine what downstream effects might occur from this change. Testing further manually, I observe some really crazy behaviour. Using a regular browser, I can keep reloading the same record over and over without hitting any limit. Using a command-line client (curl or OpenBSD ftp(1)), i hit some extremely strict limits very quickly. On the 11th lookup, I receiving this, and it started counting down: {"message": "Request was throttled. Expected available in 59 minutes. Authenticate for less restrictions. For more information: https://docs.peeringdb.com/howto/api_keys/", "meta": {"error": "Too Many Requests"}} All requests are blocked for an hour. I paused, and tried a new lookup every 5 minutes. Half an hour later it allowed me to retrieve 1 record, but then continued the countdown: {"message": "Request was throttled. Expected available in 17 minutes. Authenticate for less restrictions. For more information: https://docs.peeringdb.com/howto/api_keys/", "meta": {"error": "Too Many Requests"}} If the new limit is 10 records an hour, I mean you could really just shut down the service entirely, there's really no difference. I understand using APIKEY is the new way, but the old way has effectively been disabled without notice. But also, why does this work in the browser? I can reload hundreds of times. Is the official browser User-agent being considered exempt from the rule? If so I don't understand what the protection plan is.. maybe we should all change our commandline tools to declare they are Chrome?

5 6

peeringdb-py example setup
by Chris Caputo 19 Sep '22

19 Sep '22

In case useful to others, below is how peeringdb-py is set up & upgraded at the SeattleIX. Questions/improvements welcome. Chris --- INSTALL: mkdir ~/peeringdb-py && cd ~/peeringdb-py virtualenv --python=python3 pdbvenv source pdbvenv/bin/activate pip install --upgrade pip pip install -U setuptools pip install git+https://github.com/peeringdb/django-peeringdb.git pip install git+https://github.com/peeringdb/peeringdb-py.git pip install "django<3" Write defaults: peeringdb config set -n # Adjust [HOME] below to be the home directory. edit ~/.peeringdb/config.yaml change: name: /home/[HOME]/peeringdb-py/peeringdb.sqlite3 change: api_key: '[Read-only Org/User API Key]' peeringdb config show | sed 's/ api_key:.*/ api_key: CENSORED/g' orm: backend: django_peeringdb database: engine: sqlite3 host: '' name: /home/[HOME]/peeringdb-py/peeringdb.sqlite3 password: '' port: 0 user: '' migrate: true secret_key: '' sync: api_key: CENSORED only: [] password: '' strip_tz: 1 timeout: 0 url: https://www.peeringdb.com/api user: '' peeringdb --version peeringdb 1.2.1.1 django-admin --version 2.2.28 pip freeze | grep django-peeringdb django-peeringdb @ git+https://github.com/peeringdb/django-peeringdb.git@14eb990d6803e89f38027… time peeringdb sync real 14m47.515s user 14m27.077s sys 0m1.939s time peeringdb sync real 0m3.110s user 0m1.074s sys 0m0.088s edit ~/crontab add: # Run hourly with a random 5 minute sleep at start, to reduce thundering herd load on PeeringDB servers add: 00 * * * * sleep $[RANDOM\%300] ; cd /home/[HOME]/peeringdb-py ; touch peeringdb.sync.log ; date >> peeringdb.sync.log ; ./pdbvenv/bin/peeringdb sync >> peeringdb.sync.log 2>&1 crontab ~/crontab crontab -l # confirm above setting of crontab --- UPGRADE: cd ~/peeringdb-py tar cvzf pdbvenv.`date +\%Y\%m\%d\%H\%M\%S`.tgz pdbvenv # make a backup source pdbvenv/bin/activate pip install --upgrade pip pip install -U setuptools pip install git+https://github.com/peeringdb/django-peeringdb.git pip install --upgrade git+https://github.com/peeringdb/peeringdb-py.git pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U pip install "django<3" peeringdb --version peeringdb 1.2.1.1 django-admin --version 2.2.28 pip freeze | grep django-peeringdb django-peeringdb @ git+https://github.com/peeringdb/django-peeringdb.git@14eb990d6803e89f38027… time peeringdb sync real 0m2.809s user 0m0.772s sys 0m0.045s --- EXAMPLE USAGE: cd ~/peeringdb-py sqlite3 peeringdb.sqlite3 .schema # dump schema sqlite3 -header peeringdb.sqlite3 "SELECT * FROM peeringdb_network_ixlan WHERE asn = 6456;" # show AS6456 id|status|created|updated|version|asn|ipaddr4|ipaddr6|is_rs_peer|notes|speed|ixlan_id|net_id|operational 1534|ok|2010-07-29 00:00:00|2020-10-09 20:54:19|0|6456|206.81.80.10|2001:504:16::1938|1||20000|13|416|1 21681|ok|2015-02-06 00:00:00|2020-10-09 20:54:20|0|6456|206.81.81.41|2001:504:16::297:0:1938|1||20000|13|416|1 28528|ok|2016-04-22 03:19:39|2020-10-09 20:54:19|0|6456|206.81.82.10|2001:504:16:1::1938|1||20000|1285|416|1 28529|ok|2016-04-22 03:20:00|2020-10-09 20:54:20|0|6456|206.81.83.41|2001:504:16:1:0:297:0:1938|1||20000|1285|416|1 ---

1 0

PeeringDB Account Verification
by Noelle Kenny 19 Sep '22

19 Sep '22

Hello, My name is Noelle Kenny with Inflect. I'm hoping to create a verified account with PeeringDB in order to gain access to the API. I don't have an organization in PeeringDB to affiliate with, which may be why I'm unable to be verified. Any insight would be greatly appreciated. Thanks, Noelle -- *Noelle Kenny *| *Inflect, Inc. * Product Manager Noelle(a)inflect.com | *818-645-0325* *LinkedIn* <https://www.linkedin.com/in/noelle-kenny-b250a21b/> *Discord: *Noelle | FAB#7684 <https://inflect.com/>

5 5