Unable to reproduce the issue using ArcGIS Enterprise 11.5. To achieve the best performance for a geoprocessing service solving a vehicle routing problem, use the arcpy.nax API to develop the script tool instead. For more information, refer to ArcGIS Pro: VehicleRoutingProblem (https://pro.arcgis.com/en/pro-app/latest/arcpy/network-analyst/vehicleroutingproblem.htm).
Alternatively, publish directly to ArcGIS Server or ArcGIS Enterprise using the publish routing services workflow with own data. For more information, refer to ArcGIS Server: Publish routing services (https://enterprise.arcgis.com/en/server/latest/publish-services/windows/publish-routing-services.htm).
Services published using the publish routing services workflow are written with arcpy.nax and perform well.