[PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
Brian Dickson
brian.peter.dickson at gmail.com
Fri May 15 14:55:35 PDT 2020
Yes, it's still running, but I'm not getting the error messages I was
seeing before.
I expect it to work completely now.
Thanks, everyone, especially Milton!
(Also, you may want to approve his patch. :-) )
Brian
On Fri, May 15, 2020 at 2:42 PM Chris Caputo <ccaputo at alt.net> wrote:
> Any better with Milton's 'self.start_time = datetime.utcnow()' change?
>
> Thanks,
> Chris
>
> On Fri, 15 May 2020, Brian Dickson wrote:
> > On Fri, May 15, 2020 at 12:57 PM Brian Dickson <
> brian.peter.dickson at gmail.com> wrote:
> > The local database is empty. I'll make sure that's the case just
> to be safe.
> >
> > # peeringdb --version
> >
> > peeringdb 1.0.0
> >
> >
> > # django-admin --version
> >
> > 2.2.12
> >
> >
> > # pip freeze
> >
> > asgiref==3.2.7
> >
> > certifi==2020.4.5.1
> >
> > cfu==1.5.0
> >
> > chardet==3.0.4
> >
> > click==7.1.2
> >
> > Django==2.2.12
> >
> > django-countries==6.1.2
> >
> > django-handleref==0.5.0
> >
> > django-inet==0.4.0
> >
> > django-peeringdb==2.0.0
> >
> > future==0.18.2
> >
> > idna==2.9
> >
> > munge==1.0.0
> >
> > mysqlclient==1.4.4
> >
> > peeringdb==1.0.0
> >
> > PyMySQL==0.9.3
> >
> > pytz==2020.1
> >
> > PyYAML==5.3.1
> >
> > requests==2.23.0
> >
> > six==1.14.0
> >
> > sqlparse==0.3.1
> >
> > twentyc.rpc==0.4.0
> >
> > urllib3==1.25.9
> >
> >
> >
> > Here is the result when running the sync on a clean empty local database:
> >
> > bash-3.2# peeringdb drop-tables
> >
> > bash-3.2# peeringdb sync
> >
> > Syncing to https://www.peeringdb.com/api
> >
> > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
> >
> > Fetching & updating all: org
> >
> > Updates to be processed: 18439
> >
> > Ignoring object updated after sync began: (org-6864)
> >
> > Ignoring object updated after sync began: (org-13687)
> >
> > Ignoring object updated after sync began: (org-23582)
> >
> > Ignoring object updated after sync began: (org-26092)
> >
> > Ignoring object updated after sync began: (org-26094)
> >
> > Ignoring object updated after sync began: (org-26095)
> >
> > Ignoring object updated after sync began: (org-26096)
> >
> > Ignoring object updated after sync began: (org-26097)
> >
> > Ignoring object updated after sync began: (org-26098)
> >
> > Fetching & updating all: fac
> >
> > Updates to be processed: 3714
> >
> > Ignoring object updated after sync began: (fac-3634)
> >
> > Ignoring object updated after sync began: (fac-8262)
> >
> > Ignoring object updated after sync began: (fac-8582)
> >
> > Ignoring object updated after sync began: (fac-8583)
> >
> > Ignoring object updated after sync began: (fac-8584)
> >
> > Ignoring object updated after sync began: (fac-8585)
> >
> > Ignoring object updated after sync began: (fac-8586)
> >
> > Ignoring object updated after sync began: (fac-8587)
> >
> > Ignoring object updated after sync began: (fac-8588)
> >
> > Ignoring object updated after sync began: (fac-8591)
> >
> > Ignoring object updated after sync began: (fac-8592)
> >
> > Ignoring object updated after sync began: (fac-8593)
> >
> > Ignoring object updated after sync began: (fac-8594)
> >
> > Ignoring object updated after sync began: (fac-8595)
> >
> > Ignoring object updated after sync began: (fac-8597)
> >
> > Ignoring object updated after sync began: (fac-8598)
> >
> > Ignoring object updated after sync began: (fac-8599)
> >
> > Ignoring object updated after sync began: (fac-8600)
> >
> > Ignoring object updated after sync began: (fac-8601)
> >
> > Ignoring object updated after sync began: (fac-8602)
> >
> > Ignoring object updated after sync began: (fac-8603)
> >
> > Fetching & updating all: net
> >
> > Updates to be processed: 18845
> >
> > Ignoring object updated after sync began: (net-2913)
> >
> > Ignoring object updated after sync began: (net-3087)
> >
> > Ignoring object updated after sync began: (net-4823)
> >
> > Ignoring object updated after sync began: (net-5442)
> >
> > Ignoring object updated after sync began: (net-7482)
> >
> > Ignoring object updated after sync began: (net-9435)
> >
> > Ignoring object updated after sync began: (net-13681)
> >
> > Ignoring object updated after sync began: (net-14816)
> >
> > Ignoring object updated after sync began: (net-19500)
> >
> > Ignoring object updated after sync began: (net-20575)
> >
> > Ignoring object updated after sync began: (net-21292)
> >
> > Ignoring object updated after sync began: (net-21916)
> >
> > Ignoring object updated after sync began: (net-23222)
> >
> > Ignoring object updated after sync began: (net-23266)
> >
> > Ignoring object updated after sync began: (net-23280)
> >
> > Ignoring object updated after sync began: (net-23283)
> >
> > Ignoring object updated after sync began: (net-23284)
> >
> > Ignoring object updated after sync began: (net-23285)
> >
> > Traceback (most recent call last):
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb",
> line 8, in <module>
> >
> > sys.exit(main())
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
> line 62, in main
> >
> > return handler(config=cfg, **vars(options))
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
> line 20, in _wrapped
> >
> > r = func(*a, **k)
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
> line 219, in handle
> >
> > client.update_all(rs)
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 66, in update_all
> >
> > self._atomic_update(lambda: ctx.sync_resource(r, since=since))
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 78, in _atomic_update
> >
> > sync_func()
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 66, in <lambda>
> >
> > self._atomic_update(lambda: ctx.sync_resource(r, since=since))
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
> line 65, in _wrapped
> >
> > return loop.run_until_complete(func(*a, **k))
> >
> > File
> "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
> line 488, in run_until_complete
> >
> > return future.result()
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
> line 41, in _wrapped
> >
> > item = gen.send(r)
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 287, in sync_row
> >
> > B.clean(obj)
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend.py",
> line 145, in clean
> >
> > obj.full_clean()
> >
> > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
> line 1203, in full_clean
> >
> > raise ValidationError(errors)
> >
> > django.core.exceptions.ValidationError: {'org': ['organization instance
> with id 13687 does not exist.']}
> >
> >
> >
> > On Fri, May 15, 2020 at 10:55 AM Chris Caputo <ccaputo at alt.net>
> wrote:
> > Hi. Does restarting with a fresh/empty local database help at all?
> >
> > Also, what does the following show?:
> >
> > peeringdb --version
> > django-admin --version
> > pip freeze
> >
> > Thanks,
> > Chris
> >
> > On Fri, 15 May 2020, Brian Dickson wrote:
> > > Arnold Nipper said to send to this list.
> > >
> > > (This is kind of urgent for me, my goal is to get a snapshot of
> the data, so any alternative workaround would be helpful in the
> > > meantime.)
> > >
> > > I'm in the process of setting up a local copy, and have run into
> a problem after getting the peeringdb-py stuff set up.
> > >
> > > When I do peeringdb sync (with the correct mysql database UTF8
> stuff and everything,), I am currently getting a persistent error
> > > which aborts the sync with no data in the tables:
> > >
> > > (Apologies in advance for the long error output text.)
> > > Syncing to https://www.peeringdb.com/api
> > > Updating resources: org fac net ix ixfac ixlan ixpfx netfac
> netixlan poc
> > > Fetching & updating all: org
> > > Updates to be processed: 5
> > > Ignoring object updated after sync began: (org-26015)
> > > Ignoring object updated after sync began: (org-26052)
> > > Ignoring object updated after sync began: (org-17918)
> > > Ignoring object updated after sync began: (org-26053)
> > > Fetching & updating all: fac
> > > Updates to be processed: 0
> > > Fetching & updating all: net
> > > Updates to be processed: 18800
> > > Ignoring object updated after sync began: (net-1356)
> > > Ignoring object updated after sync began: (net-3684)
> > > Ignoring object updated after sync began: (net-7924)
> > > Ignoring object updated after sync began: (net-10733)
> > > Ignoring object updated after sync began: (net-13084)
> > > Ignoring object updated after sync began: (net-14581)
> > > Ignoring object updated after sync began: (net-15702)
> > > Traceback (most recent call last):
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
> > > line 84, in _execute
> > > return self.cursor.execute(sql, params)
> > > File
> > >
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py",
> line
> > > 71, in execute
> > > return self.cursor.execute(query, args)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
> line 209, in
> > > execute
> > > res = self._query(query)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
> line 315, in
> > > _query
> > > db.query(q)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py",
> line
> > > 226, in query
> > > _mysql.connection.query(self, query)
> > > MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update
> a child row: a foreign key constraint fails
> > > (`peeringdb`.`peeringdb_network`, CONSTRAINT
> `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY
> > > (`org_id`) REFERENCES `peeringdb_organization` (`id`))')
> > >
> > > The above exception was the direct cause of the following
> exception:
> > >
> > > Traceback (most recent call last):
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb",
> line 8, in <module>
> > > sys.exit(main())
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
> line 62, in
> > > main
> > > return handler(config=cfg, **vars(options))
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
> line 20,
> > > in _wrapped
> > > r = func(*a, **k)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
> line 219,
> > > in handle
> > > client.update_all(rs)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 66,
> > > in update_all
> > > self._atomic_update(lambda: ctx.sync_resource(r,
> since=since))
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 78,
> > > in _atomic_update
> > > sync_func()
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 66,
> > > in <lambda>
> > > self._atomic_update(lambda: ctx.sync_resource(r,
> since=since))
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
> line
> > > 65, in _wrapped
> > > return loop.run_until_complete(func(*a, **k))
> > > File
> "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
> line 488, in
> > > run_until_complete
> > > return future.result()
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
> line
> > > 41, in _wrapped
> > > item = gen.send(r)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 274,
> > > in sync_row
> > > B.save(obj)
> > >
> File"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend
> > > .py", line 148, in save
> > > obj.save()
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
> line
> > > 741, in save
> > > force_update=force_update, update_fields=update_fields)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
> line
> > > 779, in save_base
> > > force_update, using, update_fields,
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
> line
> > > 870, in _save_table
> > > result = self._do_insert(cls._base_manager, using, fields,
> update_pk, raw)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
> line
> > > 908, in _do_insert
> > > using=using, raw=raw)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/manager.py",
> > > line 82, in manager_method
> > > return getattr(self.get_queryset(), name)(*args, **kwargs)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/query.py",
> line
> > > 1186, in _insert
> > > return query.get_compiler(using=using).execute_sql(return_id)
> > > File
> > >
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py",
> line
> > > 1375, in execute_sql
> > > cursor.execute(sql, params)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
> > > line 67, in execute
> > > return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
> > > line 76, in _execute_with_wrappers
> > > return executor(sql, params, many, context)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
> > > line 84, in _execute
> > > return self.cursor.execute(sql, params)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/utils.py",
> line 89, in
> > > __exit__
> > > raise dj_exc_value.with_traceback(traceback) from exc_value
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
> > > line 84, in _execute
> > > return self.cursor.execute(sql, params)
> > > File
> > >
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py",
> line
> > > 71, in execute
> > > return self.cursor.execute(query, args)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
> line 209, in
> > > execute
> > > res = self._query(query)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
> line 315, in
> > > _query
> > > db.query(q)
> > > File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py",
> line
> > > 226, in query
> > > _mysql.connection.query(self, query)
> > > django.db.utils.IntegrityError: (1452, 'Cannot add or update a
> child row: a foreign key constraint fails
> > > (`peeringdb`.`peeringdb_network`, CONSTRAINT
> `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY
> > > (`org_id`) REFERENCES `peeringdb_organization` (`id`))')
> > >
> > > Please let me know if this is a problem on my end, or a problem
> with the data in the master db itself...
> > >
> > > If there is something I should do to fix this on my end, any
> advice would be greatly appreciated.
> > > (If there is any other way to grab a copy of the tables in mysql
> format, that's all I'm really trying to get, just a snapshot of
> > > the current peeringdb database to process with some perl scripts
> I have.)
> > >
> > > Thanks in advance,
> > >
> > > Brian Dickson
> > >
> > >
> > >
> >
> >
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.peeringdb.com/pipermail/pdb-tech/attachments/20200515/c535d42b/attachment-0001.htm>
More information about the Pdb-tech
mailing list