[PDB Data Ownership-TF] IXP assignment IP address (netixlan) ownership
Chris Caputo
ccaputo at alt.net
Tue Jan 28 19:33:01 PST 2020
On Tue, 28 Jan 2020, William Marantz wrote:
> On Tue, Jan 28, 2020 at 2:27 AM Arnold Nipper <arnold at peeringdb.com> wrote:
> > On 27.01.2020 18:37, Chris Caputo wrote:
> > > [...] what I perceive you believe is that it is okay to publicize
> > > what may be incorrect data, rather than resolve the conflict first.
> > >
> > > Just as IX-F JSON data can be wrong, so can Network-provided
> > > netixlan data be wrong. I argue that when there is a difference, it
> > > is best to play it safe and not publicize until resolved.
> > >
> > > If you are concerned about what was previously unconflicted data
> > > being pulled down by accident, then maybe the handling of previously
> > > published data should be different than the publishing of new data?
> > > Could that be the solution we are looking for? Ex:
> > >
> > > if (data is in conflict)
> > > {
> > > if (already published)
> > > {
> > > keep data up
> > > initiate resolution process
> > > }
> > > else
> > > {
> > > don't put data up
> > > initiate resolution process
> > > }
> > > }
> > >
> > > On Sat, 25 Jan 2020, Arnold Nipper wrote:
> > > > Even though we consider data from IXP as authoritative, it might
> > > > contain errors. According to your proposal data would disappear in
> > > > the API. Booom!
> > >
> > > I think the above avoids the "Booom!", if "Booom!" refers to
> > > unintentionally causing deprovisioning.
> >
> > Now it starts to make sense to me as well. If you could also take #539
> > into account we should have a made a huge step forward.
>
> I support that "already published" modification to your proposal Chris.
> I totally forgot about that use case and great idea.
Excellent! Ok, I spent a bunch of time today trying to pull #539, #585,
and #627 together in some coherent way. Please review the below and let
me know of mistakes. Maybe we can beat this around into something useful.
Thanks!
Chris
// Pseudocode for the following GitHub issues:
//
// - Add attribute `operational` to `netixlan` #539
// - Interim IX-F JSON importer #585
// - netixlan conflict resolution idea #627
//
// Related tables:
//
// - ixlan:
// - TODO gains 'ixf_ixp_import_last_timestamp' in appropriate
// format (ex. unixtime or whatever)
//
// - netixlan:
// - existing table contains data supplied by Networks or
// IXP IX-F JSON import if "Allow IXP Update" enabled.
// - TODO gains 'operational' boolean
// - TODO gains 'published' boolean
//
// - netixlan_ixp:
// - TODO new table which is a clone of netixlan's schema
// - TODO contains data from IXP IX-F JSON imports
// - TODO updated hourly via cron task which also takes care of
// "Allow IXP Update" tasks.
//
// - netixlan_override:
// - TODO new table which is a clone of netixlan's schema
// - contains Admin Committee overrides
//
// PRE-REQUISITE PRIOR TO FIRST RUN:
//
// - All currently active (status == 'ok') netixlan records need
// to have both 'operational' and 'published' set to TRUE.
//
#define IXF_SECONDS_UNTIL_STALE 30*86400 // to be tuned
// Run as a cron task, likely hourly, and/or based on to-be-designed push-to-pull-trigger-mechanism.
ixf_json_importer_per_ixlan(ixlan ixp)
{
// Mark all netixlan_ixp records for this IXP for pending deletion,
// in case they aren't in current update.
for_each (record in ixp.netixlan_ixp)
{
ixp.netixlan_ixp[record].pending_deletion = TRUE;
}
// Review each entry in the IX-F JSON export.
for_each ([asn][ipv4][ipv6] in ixf_export)
{
record = [asn][ipv4][ipv6];
// Update netixlan_ixp accordingly.
ixp.netixlan_ixp[record].asn = asn;
ixp.netixlan_ixp[record].ipv4 = ipv4;
ixp.netixlan_ixp[record].ipv6 = ipv6;
ixp.netixlan_ixp[record].pending_deletion = FALSE;
if (ixf_export[record].status == something_that_means_operational)
{
ixp.netixlan_ixp[record].operational = TRUE;
}
else
{
ixp.netixlan_ixp[record].operational = FALSE;
}
// Consider netixlan[record] update.
if (network[asn].Allow_ixp_update == no)
{
// From #585:
// If a network has an IXP entry with differing (asn, ipaddr4, ipaddr6),
// create a DeskPRO ticket, emailing the network and the ix. (1)
//
// If a network has an IXP entry with any other differing information
// (speed, route server peer), this information is not changed
//
// If a network does not have an entry for the IXP, nothing is done
if (conflict)
{
// Per the above: create a DeskPRO ticket, email network and ix
if (NOT ticket_for_this_triple(asn,ipv4,ipv6))
{
ticket();
}
}
}
else
{
// network[asn].Allow_ixp_update == yes
// From #585:
// If a network has an IXP entry with any differing information, the
// entry is updated (IPv4, IPv6, speed, route server peer) (1)
//
// If a network does not have an entry for the IXP,
// - if there is not a conflicting record, a new entry is added.
// - If there is a conflicting record, create a DeskPRO ticket,
// emailing the networks and the ix.
//
// (1) This also covers the case that the network does not have an
// entry in the IXP member list at all. In the case of
// "allow_ixp_update: yes" the update then actually means removal.
// This prevents involved parties from being flooded by tickets
//
// A DeskPRO ticket will be opened once only and thereby creates a
// lock. Only if the lock is removed a new ticket can be created.
if (no conflict)
{
// Network's netixlan gets a copy of the IXP's netixlan_ixp.
ixp.netixlan[record] = ixp.netixlan_ixp[record];
}
else
{
// Per the above: create a DeskPRO ticket, email network and ix
if (NOT ticket_for_this_triple(asn,ipv4,ipv6))
{
ticket();
}
}
}
}
// Delete records not found in most recent parsing of IX-F JSON export.
for_each (record in ixp.netixlan_ixp)
{
if (ixp.netixlan_ixp[record].pending_deletion == TRUE)
{
delete(ixp.netixlan_ixp[record]);
}
}
// Success, so update timestamp.
ixlan.ixf_ixp_import_last_timestamp = now();
// If for some reason there is not success, create a ticket and notify the IXP
// that their data is failing to be parsed.
if (NOT ticket_for_this_ixp(ixp))
{
ticket();
}
}
// Run when a Network makes an update to their netixlan record via the web site or the API.
network_update(ixlan ixp,
int asn,
cidr ipv4,
cidr ipv6)
{
// New field "operational" defaults to TRUE in the web user interface.
// Save the update.
ixp.netixlan[asn][ipv4][ipv6] = UPDATES;
// Call display_handler() so Network quickly gets alerted to any issues with their update.
// All of this ASN's records at the IXP are evaluated.
display_handler(ixp, asn);
}
// Run when the API or web site considers displaying a netixlan record, or when there has been an update.
display_handler(ixlan ixp,
int asn)
{
list blocklist[]; // contains ipv4 and ipv6 addresses that have already been considered
// Go through Admin Committee overrides first...
for_each (record in ixp.netixlan_override[asn])
{
blocklist[].append(ixp.netixlan_override[record].ipv4);
blocklist[].append(ixp.netixlan_override[record].ipv6);
if (ixp.netixlan_override[record].operational == TRUE)
{
// Publish the record, if operational, either by displaying on web pages (/ix/, /net/)
// or including in API results.
ixp.netixlan_override[record].published = TRUE;
publish(ixp.netixlan_override[record]);
}
}
// Go through Network provided entries...
for_each (record in ixp.netixlan[asn])
{
ipv4 = ixp.netixlan[record].ipv4;
ipv6 = ixp.netixlan[record].ipv6;
if (blocklist[].contains(ipv4) || blocklist[].contains(ipv6))
{
// Admin Committee's ixp.netixlan_override already dealt with this IP addr.
continue;
}
blocklist[].append(ixp.netixlan[record].ipv4);
blocklist[].append(ixp.netixlan[record].ipv6);
// Skip if this is not an operational record.
if (ixp.netixlan[record].operational == FALSE)
{
// Update 'published' accordingly, in case network is going on hiatus.
ixp.netixlan[record].published == FALSE;
continue;
}
if (ixp.ixf_ixp_import_enabled == TRUE &&
ixp.ixf_ixp_import_last_timestamp + IXF_SECONDS_UNTIL_STALE > NOW)
{
// We have recent IXP IX-F JSON records to consider...
// Do we have a matching IXP provided netixlan_ixp entry?
if (ixp.netixlan_ixp[asn][ipv4][ipv6] &&
ixp.netixlan_ixp[asn][ipv4][ipv6].operational == TRUE)
{
// Yes, IX-F matches Network provided...
// Publish the record, either by displaying on web pages (/ix/, /net/)
// or including in API results.
ixp.netixlan[record].published = TRUE;
publish(ixp.netixlan[record]);
// If ticket previously created for this triple, update it to say the
// conflict was resolved.
if (ticket_for_this_triple(asn,ipv4,ipv6))
{
ticket_update();
}
}
else
{
// No, IX-F does not match Network provided info... Conflict!
// Is this already published data?
if (ixp.netixlan[record].published == TRUE)
{
// We need to continue to publish it, since it has already been published,
// to avoid accidental disruption of provisioning systems, but raise alerts.
publish(ixp.netixlan[record]);
// Create a DeskPRO ticket, email network and ix.
if (NOT ticket_for_this_triple(asn,ipv4,ipv6))
{
ticket();
}
}
else
{
// Is not currently published. Meaning this is data which is not found
// in the IXP IX-F JSON export and thus should not be made public in such a way
// as to cause provisioning. Rather indicate it only on the Network's /net/ page
// with a conflict indication which gracefully highlights the conflict in such a way
// that the network admins and any visitors know it is in conflict with IXP data.
net_web_page_show_with_conflict_indications(ixp.netixlan[record]);
// Create a DeskPRO ticket, email network and ix.
if (NOT ticket_for_this_triple(asn,ipv4,ipv6))
{
ticket();
}
}
}
}
else
{
// IXP IX-F JSON data is stale or non-existant.
// Publish the record, either by displaying on web pages (/ix/, /net/)
// or including in API results.
ixp.netixlan[record].published = TRUE;
publish(ixp.netixlan[record]);
}
}
// Go through unmatched IXP IX-F JSON provided entries...
for_each (record in ixp.netixlan_ixp[asn])
{
ipv4 = ixp.netixlan_ixp[record].ipv4;
ipv6 = ixp.netixlan_ixp[record].ipv6;
if (blocklist[].contains(ipv4) || blocklist[].contains(ipv6))
{
// Admin Committee's ixp.netixlan_override already dealt with this IP addr.
continue;
}
// Present message only to network's admin users when they view their /net/ page
// indicating an IP assignment they are missing which they could add or dismiss.
// Remind them of the IX-F auto-import option and preview.
net_web_page_offer_ixf_hint_to_admins(ixp.netixlan_ixp[record]);
}
}
More information about the DataOwnership-TF
mailing list