[PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

Brian Dickson brian.peter.dickson at gmail.com
Fri May 15 13:08:43 PDT 2020


On Fri, May 15, 2020 at 12:57 PM Brian Dickson <
brian.peter.dickson at gmail.com> wrote:

> The local database is empty. I'll make sure that's the case just to be
> safe.
>
> # peeringdb --version
>
> peeringdb 1.0.0
>
>
> # django-admin --version
>
> 2.2.12
>
>
> # pip freeze
>
> asgiref==3.2.7
>
> certifi==2020.4.5.1
>
> cfu==1.5.0
>
> chardet==3.0.4
>
> click==7.1.2
>
> Django==2.2.12
>
> django-countries==6.1.2
>
> django-handleref==0.5.0
>
> django-inet==0.4.0
>
> django-peeringdb==2.0.0
>
> future==0.18.2
>
> idna==2.9
>
> munge==1.0.0
>
> mysqlclient==1.4.4
>
> peeringdb==1.0.0
>
> PyMySQL==0.9.3
>
> pytz==2020.1
>
> PyYAML==5.3.1
>
> requests==2.23.0
>
> six==1.14.0
>
> sqlparse==0.3.1
>
> twentyc.rpc==0.4.0
>
> urllib3==1.25.9
>


Here is the result when running the sync on a clean empty local database:

bash-3.2# peeringdb drop-tables

bash-3.2# peeringdb sync

Syncing to https://www.peeringdb.com/api

Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc

Fetching & updating all: org

Updates to be processed: 18439

Ignoring object updated after sync began: (org-6864)

Ignoring object updated after sync began: (org-13687)

Ignoring object updated after sync began: (org-23582)

Ignoring object updated after sync began: (org-26092)

Ignoring object updated after sync began: (org-26094)

Ignoring object updated after sync began: (org-26095)

Ignoring object updated after sync began: (org-26096)

Ignoring object updated after sync began: (org-26097)

Ignoring object updated after sync began: (org-26098)

Fetching & updating all: fac

Updates to be processed: 3714

Ignoring object updated after sync began: (fac-3634)

Ignoring object updated after sync began: (fac-8262)

Ignoring object updated after sync began: (fac-8582)

Ignoring object updated after sync began: (fac-8583)

Ignoring object updated after sync began: (fac-8584)

Ignoring object updated after sync began: (fac-8585)

Ignoring object updated after sync began: (fac-8586)

Ignoring object updated after sync began: (fac-8587)

Ignoring object updated after sync began: (fac-8588)

Ignoring object updated after sync began: (fac-8591)

Ignoring object updated after sync began: (fac-8592)

Ignoring object updated after sync began: (fac-8593)

Ignoring object updated after sync began: (fac-8594)

Ignoring object updated after sync began: (fac-8595)

Ignoring object updated after sync began: (fac-8597)

Ignoring object updated after sync began: (fac-8598)

Ignoring object updated after sync began: (fac-8599)

Ignoring object updated after sync began: (fac-8600)

Ignoring object updated after sync began: (fac-8601)

Ignoring object updated after sync began: (fac-8602)

Ignoring object updated after sync began: (fac-8603)

Fetching & updating all: net

Updates to be processed: 18845

Ignoring object updated after sync began: (net-2913)

Ignoring object updated after sync began: (net-3087)

Ignoring object updated after sync began: (net-4823)

Ignoring object updated after sync began: (net-5442)

Ignoring object updated after sync began: (net-7482)

Ignoring object updated after sync began: (net-9435)

Ignoring object updated after sync began: (net-13681)

Ignoring object updated after sync began: (net-14816)

Ignoring object updated after sync began: (net-19500)

Ignoring object updated after sync began: (net-20575)

Ignoring object updated after sync began: (net-21292)

Ignoring object updated after sync began: (net-21916)

Ignoring object updated after sync began: (net-23222)

Ignoring object updated after sync began: (net-23266)

Ignoring object updated after sync began: (net-23280)

Ignoring object updated after sync began: (net-23283)

Ignoring object updated after sync began: (net-23284)

Ignoring object updated after sync began: (net-23285)

Traceback (most recent call last):

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb",
line 8, in <module>

    sys.exit(main())

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
line 62, in main

    return handler(config=cfg, **vars(options))

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
line 20, in _wrapped

    r = func(*a, **k)

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
line 219, in handle

    client.update_all(rs)

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
line 66, in update_all

    self._atomic_update(lambda: ctx.sync_resource(r, since=since))

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
line 78, in _atomic_update

    sync_func()

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
line 66, in <lambda>

    self._atomic_update(lambda: ctx.sync_resource(r, since=since))

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
line 65, in _wrapped

    return loop.run_until_complete(func(*a, **k))

  File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
line 488, in run_until_complete

    return future.result()

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
line 41, in _wrapped

    item = gen.send(r)

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
line 287, in sync_row

    B.clean(obj)

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend.py",
line 145, in clean

    obj.full_clean()

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
line 1203, in full_clean

    raise ValidationError(errors)

django.core.exceptions.ValidationError: {'org': ['organization instance
with id 13687 does not exist.']}


>
> On Fri, May 15, 2020 at 10:55 AM Chris Caputo <ccaputo at alt.net> wrote:
>
>> Hi.  Does restarting with a fresh/empty local database help at all?
>>
>> Also, what does the following show?:
>>
>> peeringdb --version
>> django-admin --version
>> pip freeze
>>
>> Thanks,
>> Chris
>>
>> On Fri, 15 May 2020, Brian Dickson wrote:
>> > Arnold Nipper said to send to this list.
>> >
>> > (This is kind of urgent for me, my goal is to get a snapshot of the
>> data, so any alternative workaround would be helpful in the
>> > meantime.)
>> >
>> > I'm in the process of setting up a local copy, and have run into a
>> problem after getting the peeringdb-py stuff set up.
>> >
>> > When I do peeringdb sync (with the correct mysql database UTF8 stuff
>> and everything,), I am currently getting a persistent error
>> > which aborts the sync with no data in the tables:
>> >
>> > (Apologies in advance for the long error output text.)
>> > Syncing to https://www.peeringdb.com/api
>> > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
>> > Fetching & updating all: org
>> > Updates to be processed: 5
>> > Ignoring object updated after sync began: (org-26015)
>> > Ignoring object updated after sync began: (org-26052)
>> > Ignoring object updated after sync began: (org-17918)
>> > Ignoring object updated after sync began: (org-26053)
>> > Fetching & updating all: fac
>> > Updates to be processed: 0
>> > Fetching & updating all: net
>> > Updates to be processed: 18800
>> > Ignoring object updated after sync began: (net-1356)
>> > Ignoring object updated after sync began: (net-3684)
>> > Ignoring object updated after sync began: (net-7924)
>> > Ignoring object updated after sync began: (net-10733)
>> > Ignoring object updated after sync began: (net-13084)
>> > Ignoring object updated after sync began: (net-14581)
>> > Ignoring object updated after sync began: (net-15702)
>> > Traceback (most recent call last):
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
>> > line 84, in _execute
>> >     return self.cursor.execute(sql, params)
>> >   File
>> >
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py",
>> line
>> > 71, in execute
>> >     return self.cursor.execute(query, args)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
>> line 209, in
>> > execute
>> >     res = self._query(query)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
>> line 315, in
>> > _query
>> >     db.query(q)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py",
>> line
>> > 226, in query
>> >     _mysql.connection.query(self, query)
>> > MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a
>> child row: a foreign key constraint fails
>> > (`peeringdb`.`peeringdb_network`, CONSTRAINT
>> `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY
>> > (`org_id`) REFERENCES `peeringdb_organization` (`id`))')
>> >
>> > The above exception was the direct cause of the following exception:
>> >
>> > Traceback (most recent call last):
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb",
>> line 8, in <module>
>> >     sys.exit(main())
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
>> line 62, in
>> > main
>> >     return handler(config=cfg, **vars(options))
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>> line 20,
>> > in _wrapped
>> >     r = func(*a, **k)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>> line 219,
>> > in handle
>> >     client.update_all(rs)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>> line 66,
>> > in update_all
>> >     self._atomic_update(lambda: ctx.sync_resource(r, since=since))
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>> line 78,
>> > in _atomic_update
>> >     sync_func()
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>> line 66,
>> > in <lambda>
>> >     self._atomic_update(lambda: ctx.sync_resource(r, since=since))
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
>> line
>> > 65, in _wrapped
>> >     return loop.run_until_complete(func(*a, **k))
>> >   File
>> "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
>> line 488, in
>> > run_until_complete
>> >     return future.result()
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
>> line
>> > 41, in _wrapped
>> >     item = gen.send(r)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>> line 274,
>> > in sync_row
>> >     B.save(obj)
>> >
>> File"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend
>> > .py", line 148, in save
>> >     obj.save()
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
>> line
>> > 741, in save
>> >     force_update=force_update, update_fields=update_fields)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
>> line
>> > 779, in save_base
>> >     force_update, using, update_fields,
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
>> line
>> > 870, in _save_table
>> >     result = self._do_insert(cls._base_manager, using, fields,
>> update_pk, raw)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
>> line
>> > 908, in _do_insert
>> >     using=using, raw=raw)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/manager.py",
>> > line 82, in manager_method
>> >     return getattr(self.get_queryset(), name)(*args, **kwargs)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/query.py",
>> line
>> > 1186, in _insert
>> >     return query.get_compiler(using=using).execute_sql(return_id)
>> >   File
>> >
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py",
>> line
>> > 1375, in execute_sql
>> >     cursor.execute(sql, params)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
>> > line 67, in execute
>> >     return self._execute_with_wrappers(sql, params, many=False,
>> executor=self._execute)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
>> > line 76, in _execute_with_wrappers
>> >     return executor(sql, params, many, context)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
>> > line 84, in _execute
>> >     return self.cursor.execute(sql, params)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/utils.py",
>> line 89, in
>> > __exit__
>> >     raise dj_exc_value.with_traceback(traceback) from exc_value
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
>> > line 84, in _execute
>> >     return self.cursor.execute(sql, params)
>> >   File
>> >
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py",
>> line
>> > 71, in execute
>> >     return self.cursor.execute(query, args)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
>> line 209, in
>> > execute
>> >     res = self._query(query)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
>> line 315, in
>> > _query
>> >     db.query(q)
>> >   File
>> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py",
>> line
>> > 226, in query
>> >     _mysql.connection.query(self, query)
>> > django.db.utils.IntegrityError: (1452, 'Cannot add or update a child
>> row: a foreign key constraint fails
>> > (`peeringdb`.`peeringdb_network`, CONSTRAINT
>> `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY
>> > (`org_id`) REFERENCES `peeringdb_organization` (`id`))')
>> >
>> > Please let me know if this is a problem on my end, or a problem with
>> the data in the master db itself...
>> >
>> > If there is something I should do to fix this on my end, any advice
>> would be greatly appreciated.
>> > (If there is any other way to grab a copy of the tables in mysql
>> format, that's all I'm really trying to get, just a snapshot of
>> > the current peeringdb database to process with some perl scripts I
>> have.)
>> >
>> > Thanks in advance,
>> >
>> > Brian Dickson
>> >
>> >
>> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.peeringdb.com/pipermail/pdb-tech/attachments/20200515/0310169a/attachment-0001.htm>


More information about the Pdb-tech mailing list