[PDB Tech] incomplete local sync

[Stefan Pratter]

Install error are caused by old pip version, can upgrade by running

pip install pip --upgrade
pip install setuptools --upgrade

install should work after that

Stefan
On 24/03/2017 20.55, Greg Dendy wrote:
> Good one, that describes my experience syncing, multiple runs needed to push through errors. My versions:
>
> django-peeringdb: 0.2.2
> peeringdb: 0.5.0
>
> Here’s how it fails on a fresh sync:
>
> https://www.peeringdb.com:443 "GET /api/net?since=0 HTTP/1.1" 200 None
> net last update 0 8933 changed
> data to be processed 8933
> Traceback (most recent call last):
>   File "/bin/peeringdb", line 11, in <module>
>     sys.exit(cli())
>   File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
>     return self.main(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
>     rv = self.invoke(ctx)
>   File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
>     return _process_result(sub_ctx.command.invoke(sub_ctx))
>   File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
>     return ctx.invoke(self.callback, **ctx.params)
>   File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
>     return callback(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/peeringdb/cli.py", line 167, in sync
>     db.sync()
>   File "/usr/lib/python2.7/site-packages/peeringdb/localdb.py", line 141, in sync
>     call_command('pdb_sync', interactive=False)
>   File "/usr/lib64/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
>     return command.execute(*args, **defaults)
>   File "/usr/lib64/python2.7/site-packages/django/core/management/base.py", line 445, in execute
>     output = self.handle(*args, **options)
>   File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 85, in handle
>     self.sync(tables)
>   File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 92, in sync
>     self.update_db(cls, self.get_objs(cls))
>   File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 127, in update_db
>     sync.sync_obj(cls, row)
>   File "/usr/lib/python2.7/site-packages/django_peeringdb/sync.py", line 25, in sync_obj
>     obj.full_clean()
>   File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", line 1171, in full_clean
>     raise ValidationError(errors)
> django.core.exceptions.ValidationError: {'info_traffic': [u"Value u'10 Tbps+' is not a valid
> choice.”]}
>
>
> So, next try to upgrade django-peeringdb. Here’s how that fails:
>
> $ pip install django-peeringdb --upgrade
> Collecting django-peeringdb
>   Using cached django-peeringdb-0.3.1.tar.gz
> Collecting django_countries>=0.1.0 (from django-peeringdb)
>   Using cached django_countries-4.2-py2.py3-none-any.whl
> Collecting django_handleref<0.2.0,>=0.1.4 (from django-peeringdb)
>   Using cached django-handleref-0.1.5.tar.gz
> Collecting django_inet<0.4,>=0.3.2 (from django-peeringdb)
>   Using cached django-inet-0.3.2.tar.gz
>     Complete output from command python setup.py egg_info:
>     /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_requires'
>       warnings.warn(msg)
>     error in django-inet setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers
>
>     ----------------------------------------
> Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-9Dm5Ax/django-inet/
>
>
>
> Greg
>
>
>> On Mar 23, 2017, at 9:49 PM, Stefan Pratter <stefan at 20c.com> wrote:
>>
>> The one situation that comes to mind that would cause what you are describing is if you are running an older version of django-peeringdb and it's failing validation on some of the enum fields that had values added to them - e.g. they are valid on the server, but not valid locally - which would affect a lot of the entities that are missing for you.
>>
>> Furthermore it would notify you of that failure initially, but consecutive runs of sync on that same database may appear to complete successfully depending on which objects they touch.
>>
>> To be sure, i just ran a local sync on a fresh database and on an existing one and it behaves normally for me in both cases.
>>
>> Tested with these versions:
>>
>> django-peeringdb==0.3.1
>> peeringdb==0.5.0
>>
>> What is the output when you run peeringdb sync on a fresh database?
>>
>> Stefan
>>
>> On 23/03/2017 23.58, Greg Dendy wrote:
>>> Hola PDB-Tech
>>>
>>> I’ve noticed that my peeringdb sql syncs over the last week or so are
>>> considerably smaller than normal, by about 75%.  The sync process
>>> appears to complete successfully, but network, facility, and IX records
>>> appear to be missing.
>>>
>>> Anyone else seeing this behavior?
>>>
>>> Are there any known issues with the sync function that could cause this
>>> behavior?
>>>
>>> Thanks,
>>>
>>> Greg
>>>
>>>
>>>
>>> _______________________________________________
>>> Pdb-tech mailing list
>>> Pdb-tech at lists.peeringdb.com
>>> http://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
>>>
>> _______________________________________________
>> Pdb-tech mailing list
>> Pdb-tech at lists.peeringdb.com
>> http://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
>