May 2020 - Pdb-tech

Re: [PDB Tech] Peeringdb sync problem
by Jan.Boogman＠swisscom.com 22 May '20

22 May '20

Hi Arnold, indeed, seems to be the same issue Cheers Jan ________________________________ Von: Arnold Nipper Gesendet: Freitag, 22. Mai 2020 12:09 Bis: Boogman Jan, INI-ONE-WSN-EIS-SNS; pdb-tech(a)lists.peeringdb.com Betreff: Re: [PDB Tech] Peeringdb sync problem Hi Jan On 22.05.2020 10:20, Jan.Boogman(a)swisscom.com wrote: > every now and then I stumble over syncing problems like this, which seem > to be caused by bad data in the db: > looks like https://github.com/peeringdb/peeringdb-py/issues/38 to me. Could you please check? Cheers Arnold -- Keep calm, keep distance, keep connected! Arnold Nipper email: arnold(a)nipper.de mobile: +49 172 2650958

3 7

Peeringdb sync problem
by Jan.Boogman＠swisscom.com 22 May '20

22 May '20

Hi every now and then I stumble over syncing problems like this, which seem to be caused by bad data in the db: pdb@pdb2:~$ peeringdb sync Syncing to https://www.peeringdb.com/api Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc Fetching & updating all: org Updates to be processed: 4 Fetching & updating all: fac Updates to be processed: 1 Fetching & updating all: net Updates to be processed: 18903 Traceback (most recent call last): File "/usr/local/bin/peeringdb", line 11, in <module> sys.exit(main()) File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/cli.py", line 62, in main return handler(config=cfg, **vars(options)) File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/commands.py", line 20, in _wrapped r = func(*a, **k) File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/commands.py", line 219, in handle client.update_all(rs) File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/_update.py", line 66, in update_all self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/_update.py", line 78, in _atomic_update sync_func() File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/_update.py", line 66, in <lambda> self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/_tasks_sequential.py", line 77, in _wrapped return _consume_task(gen) File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/_tasks_sequential.py", line 55, in _consume_task item = gen.send(r) File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/_tasks_sequential.py", line 38, in gather for r in job: File "/home/pdb/.local/lib/python2.7/site-packages/peeringdb/_update.py", line 287, in sync_row B.clean(obj) File "/home/pdb/.local/lib/python2.7/site-packages/django_peeringdb/client_adaptor/backend.py", line 145, in clean obj.full_clean() File "/home/pdb/.local/lib/python2.7/site-packages/django/db/models/base.py", line 1250, in full_clean raise ValidationError(errors) django.core.exceptions.ValidationError: {'route_server': [u'Enter a valid URL.']} pdb@pdb2:~$ Normally it fixes itself after some time (the bad data has been fixed in the db?) Is there any possibility skip this bad data? Best regards Jan

2 1

Traffic-levels adjusted
by Filiz Yilmaz 21 May '20

21 May '20

[Apologies for multiple posts] Dear all, Please note changes stemming from Github issue#519 <https://github.com/peeringdb/peeringdb/issues/519> are on beta.peeringdb.com <http://beta.peeringdb.com/> and can be tested until it is added to production during the next release, currently planned in June. With this change, traffic levels up to 100+Tbps will be added and their format will be made consistent. A diff image between beta vs production is attached to this mail. Kind regards Filiz Yilmaz PeeringDB Product Manager

1 0

Update on Data Ownership Task Force Recommendations
by Filiz Yilmaz 19 May '20

19 May '20

[Apologies for multiple posts] Dear PeeringDB Community, As previously announced last month, Data Ownership Task Force completed its work and the resulting PeeringDB Data Ownership Policy document is published at: https://docs.peeringdb.com/gov/misc/2020-04-06_PeeringDB_Data_Ownership_Pol… <https://docs.peeringdb.com/gov/misc/2020-04-06_PeeringDB_Data_Ownership_Pol…> There are some upcoming changes to PeeringDB resulting from the Task Force recommendations. Product Committee is still evaluating the implementation aspects and the planning of these changes for a future release is in progress. We will make an announcement and organise an online webinar when the planning is more concrete to better inform you about the upcoming changes. For now, you may want to have a look at the update I gave during the PeeringDB Annual Meeting on 17 April in order to get a sense of what is coming at: https://docs.peeringdb.com/presentation/20200417-DataTF-AnnualMtg-Update-Fi… <https://docs.peeringdb.com/presentation/20200417-DataTF-AnnualMtg-Update-Fi…> You may also want to keep an eye on the relevant Github issues at: • https://github.com/peeringdb/peeringdb/issues/676 <https://github.com/peeringdb/peeringdb/issues/676> • https://github.com/peeringdb/peeringdb/issues/677 <https://github.com/peeringdb/peeringdb/issues/677> • https://github.com/peeringdb/peeringdb/issues/681 <https://github.com/peeringdb/peeringdb/issues/681> Kind regards Filiz Yilmaz PeeringDB Product Manager > Begin forwarded message: > > From: Filiz Yilmaz <filiz(a)peeringdb.com> > Subject: PeeringDB Data Ownership Policy document > Date: 7 April 2020 at 17:23:31 GMT+3 > To: pdb-announce(a)lists.peeringdb.com, pdb-gov(a)lists.peeringdb.com, pdb-tech(a)lists.peeringdb.com, user-discuss(a)lists.peeringdb.com > > > Dear Peering DB Community, > > Apologies for multiple posts. > I again wanted to keep the reach of this e-mail rather wide within the Peering DB Community. > > > You may recall, back in September 2019, there was a call for volunteers to participate into Data Ownership Task Force. > This Task Force was established to discuss and agree on who owns the data tokens and/or objects in PeeringDB. The agreements, > findings and any sort of recommendations would be documented in a Policy Document as a direct outcome of the Task Force. > You can find further information about the Task Force at https://docs.peeringdb.com/taskforce/dataownership/ <https://docs.peeringdb.com/taskforce/dataownership/> > > > Data Ownership Task Force has now completed its work and the resulting PeeringDB Data Ownership Policy document is now published at: > https://docs.peeringdb.com/gov/misc/2020-04-06_PeeringDB_Data_Ownership_Pol… <https://docs.peeringdb.com/gov/misc/2020-04-06_PeeringDB_Data_Ownership_Pol…> > > You can also find the document listed on the PeeringDB Governance webpage at https://docs.peeringdb.com/gov/ <https://docs.peeringdb.com/gov/> > > > I would like to thank the Task Force members for their volunteering time, work and dedication as well as Product and Admin Committees who provided further feedback during this process. > > > In the coming weeks, Product Committee will evaluate the implementation aspects and process the changes needed on PeeringDB inline with the PeeringDB Data Ownership Policy document. > > > Stay safe and healthy! > > Kind regards > Filiz Yilmaz > PeeringDB > Product Manager > > >

1 0

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
by Brian Dickson 16 May '20

16 May '20

Arnold Nipper said to send to this list. (This is kind of urgent for me, my goal is to get a snapshot of the data, so any alternative workaround would be helpful in the meantime.) I'm in the process of setting up a local copy, and have run into a problem after getting the peeringdb-py stuff set up. When I do peeringdb sync (with the correct mysql database UTF8 stuff and everything,), I am currently getting a persistent error which aborts the sync with no data in the tables: (Apologies in advance for the long error output text.) Syncing to https://www.peeringdb.com/api Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc Fetching & updating all: org Updates to be processed: 5 Ignoring object updated after sync began: (org-26015) Ignoring object updated after sync began: (org-26052) Ignoring object updated after sync began: (org-17918) Ignoring object updated after sync began: (org-26053) Fetching & updating all: fac Updates to be processed: 0 Fetching & updating all: net Updates to be processed: 18800 Ignoring object updated after sync began: (net-1356) Ignoring object updated after sync began: (net-3684) Ignoring object updated after sync began: (net-7924) Ignoring object updated after sync began: (net-10733) Ignoring object updated after sync began: (net-13084) Ignoring object updated after sync began: (net-14581) Ignoring object updated after sync began: (net-15702) Traceback (most recent call last): File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute return self.cursor.execute(query, args) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute res = self._query(query) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query db.query(q) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query _mysql.connection.query(self, query) MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`peeringdb`.`peeringdb_network`, CONSTRAINT `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY (`org_id`) REFERENCES `peeringdb_organization` (`id`))') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", line 8, in <module> sys.exit(main()) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", line 62, in main return handler(config=cfg, **vars(options)) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 20, in _wrapped r = func(*a, **k) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 219, in handle client.update_all(rs) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in update_all self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 78, in _atomic_update sync_func() File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in <lambda> self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped return loop.run_until_complete(func(*a, **k)) File "/opt/local/Library/Frameworks/Python.framework/ Versions/3.6/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete return future.result() File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped item = gen.send(r) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 274, in sync_row B.save(obj) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend.py", line 148, in save obj.save() File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", line 741, in save force_update=force_update, update_fields=update_fields) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", line 779, in save_base force_update, using, update_fields, File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", line 870, in _save_table result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", line 908, in _do_insert using=using, raw=raw) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert return query.get_compiler(using=using).execute_sql(return_id) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1375, in execute_sql cursor.execute(sql, params) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute return self.cursor.execute(query, args) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute res = self._query(query) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query db.query(q) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query _mysql.connection.query(self, query) django.db.utils.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`peeringdb`.`peeringdb_network`, CONSTRAINT `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY (`org_id`) REFERENCES `peeringdb_organization` (`id`))') Please let me know if this is a problem on my end, or a problem with the data in the master db itself... If there is something I should do to fix this on my end, any advice would be greatly appreciated. (If there is any other way to grab a copy of the tables in mysql format, that's all I'm really trying to get, just a snapshot of the current peeringdb database to process with some perl scripts I have.) Thanks in advance, Brian Dickson

5 12