The error message, "The default toolbox does not exist" is returned for the 'Default Toolbox' path, if the Project name contains a period '.' at the second place from the left.
There are many things that can go wrong when non alphanumeric characters are included in a project's name. Each system in ArcGIS Pro has different rules regarding what characters can be included in an item's name, and what happens when those invalid characters are provided. This is one of the problems that can occur.