Gas Networks on the Move: Prepping for Smallworld to UN Migration

data transformation concept

GE Smallworld has a drastically different data model compared to Esri’s ArcGIS Utility Network and, as such, migrating from the former to the latter can involve a significant effort on behalf of your utility. However, through using specially designed migration tools and following a well-established workflow, your utility’s overall migration workload can be significantly reduced, and final data fidelity increased. 

This article sheds light on the process and challenges of migrating gas distribution and transmission data from GE Smallworld to Esri ArcGIS Utility Network, highlighting UDC’s real-world insight to help your utility understand what to expect and what you can do now to start preparing for a successful migration.

Smallworld to Utility Network Diagram
Figure 1 – Moving from GE Smallworld to Esri ArcGIS Utility Network

Overview of the Smallworld to UN Migration Process

The process of migrating from Smallworld to Utility Network (UN) involves gathering all required source data, transforming that data for consumption by Esri’s Utility and Pipeline Data Model (UPDM) and UN, and mapping and validating the data mapping through a series of steps, either manually or through an automated process, before finally creating an asset package for load into the UN target database. 

UN Migration Process Workflow
Figure 2 – UDC’s Smallworld to UN Migration Process. UDC’s auto-migration HEIDE tool provides automation for feature class and asset data mapping, attribute and topology error identification, QA/QC, containment, and advanced configuration.

Applying automation tools in your migration process to the fullest extent will save your utility significant time performing manual data and asset mapping in addition to UN error identification and resolution. Likewise, preparing for migration in advance creates a more manageable effort for your utility, by compressing the migration activities into more doable tasks.

Understanding the expectations laid out in this article will help better prepare your utility for migration, by realizing the full effort required for your unique transition. 

Mapping UPDM Feature Classes and Assets

Before the UN can be enabled for your utility, your pipeline data must first be transformed and stored in UPDM. With far fewer feature classes than Smallworld, UPDM’s data model architecture requires multiple-source feature classes to be mapped to a single-target feature class. To automate this process for our clients, we use automation tools to analyze the Smallworld data, leveraging our HEIDE tool to suggest a target location within UPDM for every source feature class and field. Following a series of reviews and corrections, the Smallworld data can then be provided in an acceptable format for load into UPDM. Automating the mapping process significantly reduces the mapping effort required by your utility.

Another consideration for your utility is that while Smallworld has separate feature classes for gas distribution and transmission data, UPDM stores both sets of data in the same feature class. Therefore, any differences between gas distribution and transmission in the Smallworld data model must be addressed, either manually or semi-automatically, during the mapping process. 

For example, Smallworld denotes “steel” as the ‘Material’ asset type for any type of steel pipe used, while the UPDM default requires an asset type distinguishment between “bare” and “coated” steel. For gas distribution data, Smallworld has an ‘External Coating’ asset field, which can be used in combination with the ‘Material’ asset field to accurately map to the UN ‘Coated Steel’ and ‘Bare Steel’ asset groups. However, the Smallworld gas transmission data has a separate feature class to represent coatings. To transpose the Smallworld transmission data into the UPDM pipeline feature class, the UPDM pipeline must be split wherever the coating type changes.

UN Data Model Architecture
Figure 3 – ArcGIS Utility Network Data Model Architecture. Ex. “PipelineDevice” feature class has an “Excess Flow Valve” asset group containing an “Automatic Reset” asset type

UN Modeling: Resolving Data Errors 

As with UPDM, the UN is designed to model real-world conditions, requiring a greater level of asset detail and accuracy from your data than typically found in Smallworld to achieve a fully connected network. For example, tracing regulator stations in UN will require an approach that ensures proper modeling and complete data for those features.

Data inaccuracies that may be present in your Smallworld GIS will cause attribute and network topology errors in the UN, preventing successful data loads. Your utility can prepare now for a combined effort of resolving undesirable errors and creating business rules for allowing exceptions to the default UN rules. Taking on this task can quickly become overwhelming for the utility. 

One solution provided by UDC for addressing error resolution is the automated analysis of your GIS source data to identify any irregularities in datasets that are not compatible with the UN’s requirements. This removes any potential human error in mapping as well as saves a substantial amount of time performing manual QC and validation. The results can then be leveraged to create an actionable data readiness plan for your utility. 

UN Modeling: Creating Business Rules

In addition to ensuring successful data loads, network rules also enforce data standards, resulting in a higher level of data fidelity for your utility. Configurable for all gas domain network and structure features, your organization can decide how restrictive to make the rules, balancing between data quality and production efficiencies. 

The following are some examples of how advanced preparation for business rule creation can benefit your utility.

Attribute Rules

Enhancing the editing experience and improving data integrity for geodatabase datasets, attribute rules are created using Geoprocessing tools or customer Arcade Scripts, such as Python. As user-defined rules, they are used to automatically populate attributes, restrict invalid edits during edit operations, and perform quality assurance checks on existing features. 

Below are the three attribute rule types within the UN:

Calculation

Immediate calculation rules automatically populate attributes based on editing patterns, increasing efficiency in data collection and editing.

Constraint

Constraint rules control data entry by user-defined requirements, preventing invalid data entry during edit operations and ensuring elevated levels of data quality in the GIS. 

Validation

Validation rules highlight errors in the existing data, allowing your utility to review features that violate specified attribute or geometry requirements. 

Attribute Rules in the UN are defined based on asset groups and asset types, representing major and minor classifications of utility features, respectively. As subcategories of feature classes, updating, adding, or removing asset groups or asset types has a trickle-down effect, requiring updates to all related attribute rules once the UN is constructed for your utility.

Therefore, your utility should permanently define asset groups and asset types prior to enabling the UN to prevent future, intensive updates. To make these decisions and provide input on data changes or data manipulations, your utility can form a data governance committee that includes stakeholders from all parts of your business, ensuring all aspects of your organization are represented in the decision process. 

To start preparations now for attribute validations, your utility can archive legacy features or attributes no longer used in the source, remove hidden attributes that are non-operational, and cleanup picklists by updating domains to remove duplicates that occur in the source database. Currently, this process can only be accomplished manually. However, through identifying and addressing potential errors early in the process, your utility can manage expectations and minimize schedule slips during migration.

Network Topology Rules

Network topology stores associations in the UN, including connectivity, containment, and structural attachment, as well as enables tracing analysis and the rapid retrieval of network features. Providing thousands of business rules related to feature connectivity, the UN defines which features can be connected and in which ways. Unlike in geometric networks where connections are determined by location, the UN is logically connected or non-coincident, meaning that connections do not require coincidence and can be spatial or non-spatial. 

Topology rules are written using the asset group or asset type descriptions and coded integer values to establish network topology. By default, UN rules are highly restrictive, as they aim to create a network that accurately models the real world through enforcing connections. A Smallworld dataset will plausibly contain a considerable number of connections that violate the default UN rules, causing errors and preventing data loads. 

Using automation, the UDC team leverages HEIDE which builds logical connectivity within the UN, analyzes errors, and generates a list of potential network topology rules that could be implemented by your utility to permit connectivity without errors. Together with our clients, we evaluate the generated topology rules to determine whether they should be added as business rules or not. Errors that are determined to be inaccuracies in the source data rather than exceptions will need to be resolved prior to loading the dataset.

Below are examples of common network topology errors we have seen that will require remediation by your utility and for which UDC has prepared tools to address:

Incomplete Regulator Stations

Since regulator stations are generally less detailed in Smallworld, the features will require proper modeling and complete data to take full advantage of tracing functionality in the UN. Regulator stations can be updated in bulk using an automation tool; however, some manual remediation may also be required in certain instances. 

Stacked Features

Multiple features that have the same geographical location will cause UN errors. These can mostly be resolved automatically by a tool; however, manual investigation may be needed when it is unclear which feature should be moved. 

Missing Data

Smallworld allows ‘Unknown’ values for some attributes that the UN does not permit. For example, it is possible to have a valve in Smallworld that has an open/close position of ‘Unknown,’ which is unacceptable in the UN. Your utility may wish to set these valves to open positions in the UN but will need to confirm the accuracy. 

Prepare Now for a Successful Migration Tomorrow

Although migrating from GE Smallworld to Esri’s ArcGIS Utility Network can be a sizable effort, your utility can start preparations now for a smoother transformation. Through using a proven workflow of mapping UPDM feature classes, creating UN business rules, and proactively resolving attribute and topology UN errors, in conjunction with migration support and tools, your utility can streamline the process of migrating its unique data and ensure a successful outcome. 

UDC’s migration experience coupled with our auto-migration tools can help your utility get started on its migration journey today. If you would like to learn more about preparing for your Utility Network migration or how HEIDE can automate your transition, we invite you to contact us.