[PDB Tech] incomplete local sync

[Greg Dendy]

Good one, that describes my experience syncing, multiple runs needed to push through errors. My versions:

django-peeringdb: 0.2.2
peeringdb: 0.5.0

Here’s how it fails on a fresh sync:

https://www.peeringdb.com:443 "GET /api/net?since=0 HTTP/1.1" 200 None
net last update 0 8933 changed
data to be processed 8933
Traceback (most recent call last):
  File "/bin/peeringdb", line 11, in <module>
    sys.exit(cli())
  File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/peeringdb/cli.py", line 167, in sync
    db.sync()
  File "/usr/lib/python2.7/site-packages/peeringdb/localdb.py", line 141, in sync
    call_command('pdb_sync', interactive=False)
  File "/usr/lib64/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/usr/lib64/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 85, in handle
    self.sync(tables)
  File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 92, in sync
    self.update_db(cls, self.get_objs(cls))
  File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 127, in update_db
    sync.sync_obj(cls, row)
  File "/usr/lib/python2.7/site-packages/django_peeringdb/sync.py", line 25, in sync_obj
    obj.full_clean()
  File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", line 1171, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'info_traffic': [u"Value u'10 Tbps+' is not a valid 
choice.”]}


So, next try to upgrade django-peeringdb. Here’s how that fails:

$ pip install django-peeringdb --upgrade
Collecting django-peeringdb
  Using cached django-peeringdb-0.3.1.tar.gz
Collecting django_countries>=0.1.0 (from django-peeringdb)
  Using cached django_countries-4.2-py2.py3-none-any.whl
Collecting django_handleref<0.2.0,>=0.1.4 (from django-peeringdb)
  Using cached django-handleref-0.1.5.tar.gz
Collecting django_inet<0.4,>=0.3.2 (from django-peeringdb)
  Using cached django-inet-0.3.2.tar.gz
    Complete output from command python setup.py egg_info:
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_requires'
      warnings.warn(msg)
    error in django-inet setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-9Dm5Ax/django-inet/



Greg


> On Mar 23, 2017, at 9:49 PM, Stefan Pratter <stefan at 20c.com> wrote:
> 
> The one situation that comes to mind that would cause what you are describing is if you are running an older version of django-peeringdb and it's failing validation on some of the enum fields that had values added to them - e.g. they are valid on the server, but not valid locally - which would affect a lot of the entities that are missing for you.
> 
> Furthermore it would notify you of that failure initially, but consecutive runs of sync on that same database may appear to complete successfully depending on which objects they touch.
> 
> To be sure, i just ran a local sync on a fresh database and on an existing one and it behaves normally for me in both cases.
> 
> Tested with these versions:
> 
> django-peeringdb==0.3.1
> peeringdb==0.5.0
> 
> What is the output when you run peeringdb sync on a fresh database?
> 
> Stefan
> 
> On 23/03/2017 23.58, Greg Dendy wrote:
>> Hola PDB-Tech
>> 
>> I’ve noticed that my peeringdb sql syncs over the last week or so are
>> considerably smaller than normal, by about 75%.  The sync process
>> appears to complete successfully, but network, facility, and IX records
>> appear to be missing.
>> 
>> Anyone else seeing this behavior?
>> 
>> Are there any known issues with the sync function that could cause this
>> behavior?
>> 
>> Thanks,
>> 
>> Greg
>> 
>> 
>> 
>> _______________________________________________
>> Pdb-tech mailing list
>> Pdb-tech at lists.peeringdb.com
>> http://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
>> 
> _______________________________________________
> Pdb-tech mailing list
> Pdb-tech at lists.peeringdb.com
> http://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech