[PDB Tech] Initial sync failing with PeeringDB Python module

[Milton Ngan]

I have noticed that the "peeringdb sync” command will fail to sync one of the tables when doing a full sync, or a large incremental sync. 

This is because while you are syncing one of the “base” tables, the dependent tables could be updated. So when you sync the dependent table, it may include records that depend on rows in the base table that weren’t inserted after your sync of the base table. 

The way to mitigate this would be to use an “until” parameter similar to the “since”. This will bound all your syncs to the same time window, ensuring a consistent update. You could probably use the “updated” field filter to achieve the same thing, but it seems inconsistent with the “since” parameter. 

This is made worse by doing incremental syncs. If a record was fails due to foreign key consistency then you only get a partial sync. It is possible that one of the fields that were sync’ed had an update timestamp way ahead of the one that failed, so the next time you sync, you end up skipping over a bunch of updates leaving gaps/inconsistencies in your data. The only way to recover from this situation is to dump your tables and perform a full, which is also unreliable for the same reasons….