Delete the older records of duplicate features after using the Append tool in ArcGIS Pro

Summary

In ArcGIS Pro, the Append tool can be used to add additional data to an existing target dataset. However, when there are features with identical locations in both the input and target datasets, running the Append tool generates duplicate features in the target dataset. It is sometimes necessary to delete duplicate features to retain only one entry per feature in the attribute table.

This article provides two workarounds to remove the older records of duplicate features from the target dataset, and only retain the newest records after running the Append tool.

Procedure

Depending on the usage, follow one of the workarounds below to solve the issue.

Use the Dissolve tool

Note:
The result of this workaround is populated in a new feature class.

1. Add a new field in the attribute table to populate with identical values to identify the duplicate features. In this example, the new field is used to identify duplicate features based on identical coordinates.

Note:
Skip Step 1 if the appended dataset has an existing field with identical values that can be used to identify the duplicate data. Ensure there are no inconsistencies between the identical values. For example, 'StreetNameA' and 'StreetName_A' are not considered identical values for this workflow.

1. In the Contents pane, right-click the feature layer containing the appended dataset, and click Attribute Table.
2. In the table view, click Add.
3. In the fields view, configure the new field. Ensure Long is selected for Data Type. In this example, the new field is named Coordinate.

4. On the Fields tab, click Save.
5. In the table view pane, right-click the header of the new field, and click Calculate Geometry.
6. In the Calculate Geometry window, for Property, select one option from the drop-down menu. In this example, Point x-coordinate is selected.
7. Click OK.

2. Use the Dissolve tool to delete the duplicate features but keep the latest appended features.
   1. On the Analysis tab, click Tools.
   2. In the Geoprocessing pane, search for and select Dissolve (Data Management Tools).
   3. In the Dissolve pane, for Input Features, select the feature class containing the appended data.
   4. For Dissolve Fields, select the field containing the identical values to identify the duplicates. In this example, Coordinate is selected.
   5. For Statistics Fields, click the down arrow next to Field, click Toggle All Checkboxes to select all fields or select the fields to preserve in the attribute table as statistics fields, and click Add.

6. For Statistic Type, select Last for all the added fields.

7. Click Run. The result of the Dissolve tool is populated in a new feature class. The duplicate features are removed and only the latest records are kept in the new feature class.

Use a custom Python script

Note:
Follow this workaround to modify the appended dataset without creating a new feature class.

When new features are appended to the target dataset, the appended features are automatically assigned a new Object ID each in sequential order following the existing data; the larger an Object ID number, the more recent a feature is added to the dataset. Therefore, the Object ID field can be used as the field to sort for the latest record by selecting the feature with the largest Object ID value among duplicate features.

Note:
Depending on the datasets used, the Object IDs are maintained in either the OBJECTID field or the OBJECTID_1 field. To identify the field storing the Object IDs, hover over the field header in the attribute table. The data type of the field storing the Object IDs is 'Object ID'. Refer to ArcMap: When is an ObjectID added to a table? for more information.

1. In ArcGIS Pro, on the Analysis tab, click the down arrow next to Python, and click Python Window.
2. Import the necessary module.

import arcpy

3. Set the path to the feature class containing the appended data, the field to search for duplicates, and the field to sort the records.

fc = r"<feature class path>"
id = "<field name>" 
sort_field = "<field name> D"

4. Define a parameter for the UpdateCursor() function and iterate through the feature class to identify and delete old duplicate records.

cursor = arcpy.UpdateCursor(fc,"","",id,sort_field)
keepList = list()
for row in cursor:
    row_val = row.getValue(id)
    if row_val not in keepList:
        keepList.append(row_val)
    elif row_val in keepList:
        cursor.deleteRow(row)
    else:
        pass

The following shows an example of the full script.

import arcpy

fc = r"C:\Users\User\Documents\ArcGIS\Projects\Append\Append.gdb\DataMain"
id = "Coordinate"
sort_field = "OBJECTID D"

cursor = arcpy.UpdateCursor(fc,"","",id,sort_field)
keepList = list()
for row in cursor:
    row_val = row.getValue(id)
    if row_val not in keepList:
        keepList.append(row_val)
    elif row_val in keepList:
        cursor.deleteRow(row)
    else:
        pass

5. Place the mouse cursor at the end of the script, and press Enter on the keyboard twice.
6. Click Refresh. The older records of duplicate features are removed from the attribute table.