After running a test upgrade I discovered that one page particular was showing up as un-ghosted (or customized) despite my setting the option to reset all pages to the site definition when I ran the upgrade. I attempted to use the browser to reset the page (in this case http://intranet/sitedirectory/lists/sites/summary.aspx) but that had no affect.

I decided that I needed to get more information about how the un-ghosting process works. The first thing I needed to do was see if there were any other pages with the same problem. To do this I created a command called gl-enumunghostedfiles. I know there are versions of the same command out there already but I found I needed something a bit more capable so that I could search an entire site collection and not just a single web.

After creating this command (detailed below) I was surprised to see that this summary.aspx page was not showing up as un-ghosted at all. When looking for an un-ghosted file you typically just check the CustomizedPageStatus property of an SPFile object. If this returns back as Customized (so much better than saying “un-ghosted” – not sure why this term came up and why I’m proliferating it :)) then the page is un-ghosted. If you look internally at how this property is evaluated you’ll see that the code is checking for the presense of a property in the Properties collection called vti_setuppath – if this property is not null or empty then it checks for another property called vti_hasdefaultcontent and if it either doesn’t find this or it’s set to false then it returns back a value of Customized. You can see this in the code below:

 1public SPCustomizedPageStatus CustomizedPageStatus
 2{
 3    get
 4    {
 5        if (!string.IsNullOrEmpty(this.SetupPath))
 6        {
 7            bool flag = false;
 8            try
 9            {
10                object obj2 = this.Properties["vti_hasdefaultcontent"];
11                if (obj2 != null)
12                {
13                    flag = bool.Parse((string) obj2);
14                }
15            }
16            catch (FormatException)
17            {
18            }
19            if (flag)
20            {
21                return SPCustomizedPageStatus.Uncustomized;
22            }
23            return SPCustomizedPageStatus.Customized;
24        }
25        return SPCustomizedPageStatus.None;
26    }
27}

What I found when I looked closer at this page that I knew (based on the shear appearance of the page) was un-ghosted was that the vti_hasdefaultcontent property was set to true and the vti_setuppath property was set to the old 2003 template path. I spent many hours trying to figure out how to re-ghost this page and ended up ultimately unsuccessful. If anyone has any thoughts on this I’d love to hear them (I’ve tried updating the SetupPath and SetupPathVerion fields in the AllDocs table to point the file to the new template but that just resulted in an unknown error when loading the page – changing the vti_hasdefaultcontent property manually also didn’t work as the value would not persist and I couldn’t find where it’s stored in the DB and changing this value in memory would allow the SPRequest.RevertContentStreams() method to be called but that would just throw a file not found exception as it is unable to locate the template file despite copying the file to various suspect locations).

As a result of my efforts though I do have a fairly robust command to re-ghost pages which does seem to work with another issue I encountered. I found that when I imported a list from another site the pages (views) for the list were showing up as un-ghosted but when I tried to use the RevertContentStream method of the SPFile object it had no affect.

After messing around with it for a while I discovered that if I called the internal SPRequest.RevertContentStreams() method directly and did not follow that call up with the call to SPRequest.UpdateFileOrFolderProperties() as the SPFile.RevertContentStream() method does then I can get the file to successfully be re-ghosted. I’ve got an email out to Microsoft to see if they can explain why this is so but in the mean-time it seems to work. The two commands that I created are detailed further below.

gl-enumunghostedfiles

The code for this command is pretty simple – I’ve basically just got two recursive methods, one for the web sites and another for folders. I allowed a parameter to be passed in to determine whether the code should recurse sub webs or not. The code is shown below:

  1public class EnumUnGhostedFiles : SPOperation
  2{
  3    /// <summary>
  4    /// Initializes a new instance of the <see cref="EnumUnGhostedFiles"/> class.
  5    /// </summary>
  6    public EnumUnGhostedFiles()
  7    {
  8        SPParamCollection parameters = new SPParamCollection();
  9        parameters.Add(new SPParam("url", "url", true, null, new SPUrlValidator(), "Please specify the site url."));
 10        parameters.Add(new SPParam("recursesubwebs", "recurse", false, null, null));
 11        Init(parameters, "\r\n\r\nReturns a list of all unghosted (customized) files for a web.\r\n\r\nParameters:\r\n\t-url <web site url>\r\n\t[-recursesubwebs]");
 12    }
 13 
 14    #region ISPStsadmCommand Members
 15 
 16    /// <summary>
 17    /// Gets the help message.
 18    /// </summary>
 19    /// <param name="command">The command.</param>
 20    /// <returns></returns>
 21    public override string GetHelpMessage(string command)
 22    {
 23        return HelpMessage;
 24    }
 25 
 26    /// <summary>
 27    /// Runs the specified command.
 28    /// </summary>
 29    /// <param name="command">The command.</param>
 30    /// <param name="keyValues">The key values.</param>
 31    /// <param name="output">The output.</param>
 32    /// <returns></returns>
 33    public override int Run(string command, System.Collections.Specialized.StringDictionary keyValues, out string output)
 34    {
 35        output = string.Empty;
 36 
 37        InitParameters(keyValues);
 38 
 39        string url = Params["url"].Value;
 40        bool recurse = Params["recursesubwebs"].UserTypedIn;
 41        List<string> unghostedFiles = new List<string>();
 42 
 43        using (SPSite site = new SPSite(url))
 44        {
 45            using (SPWeb web = site.OpenWeb())
 46            {
 47                if (recurse)
 48                {
 49                    RecurseSubWebs(web, ref unghostedFiles);
 50                }
 51                else 
 52                    CheckFoldersForUnghostedFiles(web.RootFolder, ref unghostedFiles);
 53            }
 54        }
 55 
 56        if (unghostedFiles.Count == 0)
 57        {
 58            output += "There are no unghosted (customized) files on the current web.\r\n";
 59        }
 60        else
 61        {
 62            output += "The following files are unghosted:";
 63 
 64            foreach (string fileName in unghostedFiles)
 65            {
 66                output += "\r\n\t" + fileName;
 67            }
 68        }
 69 
 70        return 1;
 71    }
 72 
 73    #endregion
 74 
 75    /// <summary>
 76    /// Recurses the sub webs.
 77    /// </summary>
 78    /// <param name="web">The web.</param>
 79    /// <param name="unghostedFiles">The unghosted files.</param>
 80    private static void RecurseSubWebs(SPWeb web, ref List<string>unghostedFiles)
 81    {
 82        foreach (SPWeb subweb in web.Webs)
 83        {
 84            try
 85            {
 86                RecurseSubWebs(subweb, ref unghostedFiles);
 87            }
 88            finally
 89            {
 90                subweb.Dispose();
 91            }
 92        }
 93        CheckFoldersForUnghostedFiles(web.RootFolder, ref unghostedFiles);
 94    }
 95 
 96    /// <summary>
 97    /// Checks the folders for unghosted files.
 98    /// </summary>
 99    /// <param name="folder">The folder.</param>
100    /// <param name="unghostedFiles">The unghosted files.</param>
101    private static void CheckFoldersForUnghostedFiles(SPFolder folder, ref List<string> unghostedFiles)
102    {
103        foreach (SPFolder sub in folder.SubFolders)
104        {
105            CheckFoldersForUnghostedFiles(sub, ref unghostedFiles);
106        }
107 
108        foreach (SPFile file in folder.Files)
109        {
110            if (file.CustomizedPageStatus == SPCustomizedPageStatus.Customized)
111            {
112                if (!unghostedFiles.Contains(file.ServerRelativeUrl))
113                    unghostedFiles.Add(file.ServerRelativeUrl);
114            }
115        }
116    }
117}

The syntax of the command can be seen below:

C:\>stsadm -help gl-enumunghostedfiles

stsadm -o gl-enumunghostedfiles

Returns a list of all unghosted (customized) files for a web.

Parameters:
        -url <web site url>
        [-recursesubwebs]

The following table summarizes the command and its various parameters:

Command NameAvailabilityBuild Date
gl-enumunghostedfilesWSS v3, MOSS 2007Released: 9/13/2007
Parameter NameShort FormRequiredDescriptionExample Usage
urlYesURL to analyze.-url http://intranet/
recursesubwebsrecurseNoIf not specified then only the single web will be considered. To recurse the web and all it’s sub-webs pass in this parameter.-recursesubwebs, -recurse

Here’s an example of how to return the un-ghosted files for a root site collection and all sub-webs:

stsadm –o gl-enumunghostedfiles -url "http://intranet/" -recursesubwebs

You can see a sample of what running the above command will produce – your results will most likely be very different:

C:\>stsadm –o gl-enumunghostedfiles -url "http://intranet/" -recursesubwebs
The following files are unghosted:
        /News/Pages/Default.aspx
        /Reports/Pages/default.aspx
        /SearchCenter/Pages/people.aspx
        /SearchCenter/Pages/default.aspx
        /SearchCenter/Pages/peopleresults.aspx
        /SearchCenter/Pages/results.aspx
        /SearchCenter/Pages/advanced.aspx
        /SiteDirectory/Pages/category.aspx
        /SiteDirectory/Pages/sitemap.aspx
        /Pages/Default.aspx
        /FormServerTemplates/Forms/InfoPath Form Template/template.doc
        /Variation Labels/NewForm.aspx
        /Variation Labels/EditForm.aspx
        /Variation Labels/AllItems.aspx
        /Variation Labels/DispForm.aspx
        /_catalogs/masterpage/VariationRootPageLayout.aspx
        /_catalogs/wp/siteFramer.dwp
        /_catalogs/wp/IViewWebPart.dwp
        /_catalogs/wp/IndicatorWebPart.dwp
        /_catalogs/wp/BusinessDataFilter.dwp
        /_catalogs/wp/KpiListWebPart.dwp
        /_catalogs/wp/SummaryLink.webpart
        /_catalogs/wp/ContentQuery.webpart
        /_catalogs/wp/ThisWeekInPictures.DWP
        /_catalogs/wp/SearchBestBets.webpart
        /_catalogs/wp/CategoryWebPart.webpart
        /_catalogs/wp/QueryStringFilter.webpart
        /_catalogs/wp/SpListFilter.dwp
        /_catalogs/wp/WSRPConsumerWebPart.dwp
        /_catalogs/wp/searchpaging.dwp
        /_catalogs/wp/contactwp.dwp
        /_catalogs/wp/searchstats.dwp
        /_catalogs/wp/UserContextFilter.webpart
        /_catalogs/wp/owacontacts.dwp
        /_catalogs/wp/SearchCoreResults.webpart
        /_catalogs/wp/BusinessDataActionsWebPart.dwp
        /_catalogs/wp/owainbox.dwp
        /_catalogs/wp/FilterActions.dwp
        /_catalogs/wp/AdvancedSearchBox.dwp
        /_catalogs/wp/BusinessDataAssociationWebPart.webpart
        /_catalogs/wp/RssViewer.webpart
        /_catalogs/wp/owatasks.dwp
        /_catalogs/wp/SearchHighConfidence.webpart
        /_catalogs/wp/PeopleSearchBox.dwp
        /_catalogs/wp/SearchActionLinks.webpart
        /_catalogs/wp/PageContextFilter.webpart
        /_catalogs/wp/owacalendar.dwp
        /_catalogs/wp/AuthoredListFilter.webpart
        /_catalogs/wp/searchsummary.dwp
        /_catalogs/wp/CategoryResultsWebPart.webpart
        /_catalogs/wp/owa.dwp
        /_catalogs/wp/OlapFilter.dwp
        /_catalogs/wp/BusinessDataDetailsWebPart.webpart
        /_catalogs/wp/TableOfContents.webpart
        /_catalogs/wp/BusinessDataListWebPart.webpart
        /_catalogs/wp/TasksAndTools.webpart
        /_catalogs/wp/Microsoft.Office.Excel.WebUI.dwp
        /_catalogs/wp/DateFilter.dwp
        /_catalogs/wp/TextFilter.dwp
        /_catalogs/wp/SearchBox.dwp
        /_catalogs/wp/BusinessDataItemBuilder.dwp
        /_catalogs/wp/TopSitesWebPart.webpart
        /_catalogs/wp/PeopleSearchCoreResults.webpart

gl-reghostfile

This command takes what should be a very simple call to SPFile.RevertContentStream() and attempts to handle those odd cases that I outlined above. Therefore the code is a bit of a mess and frankly nothing I’m proud of (mainly because I’m pissed I wasn’t able to solve the problem). The code, which uses some reflection in order to utilize some internal objects, is shown below (sorry about the poor formatting – this blog template is less than ideal for code samples):

  1/// <summary>
  2/// Runs the specified command.
  3/// </summary>
  4/// <param name="command">The command.</param>
  5/// <param name="keyValues">The key values.</param>
  6/// <param name="output">The output.</param>
  7/// <returns></returns>
  8public override int Execute(string command, System.Collections.Specialized.StringDictionary keyValues, out string output)
  9{
 10    output = string.Empty;
 11    Verbose = true;
 12    
 13 
 14    string url = Params["url"].Value;
 15    bool force = Params["force"].UserTypedIn;
 16    string scope = Params["scope"].Value.ToLowerInvariant();
 17    bool haltOnError = Params["haltonerror"].UserTypedIn;
 18 
 19    switch (scope)
 20    {
 21        case "file":
 22            using (SPSite site = new SPSite(url))
 23            using (SPWeb web = site.OpenWeb())
 24            {
 25                SPFile file = web.GetFile(url);
 26                if (file == null)
 27                {
 28                    throw new FileNotFoundException(string.Format("File '{0}' not found.", url), url);
 29                }
 30 
 31                Reghost(site, web, file, force, haltOnError);
 32            }
 33            break;
 34        case "list":
 35            using (SPSite site = new SPSite(url))
 36            using (SPWeb web = site.OpenWeb())
 37            {
 38                SPList list = Utilities.GetListFromViewUrl(web, url);
 39                ReghostFilesInList(site, web, list, force, haltOnError);
 40            }
 41            break;
 42        case "web":
 43            bool recurseWebs = Params["recursewebs"].UserTypedIn;
 44            using (SPSite site = new SPSite(url))
 45            using (SPWeb web = site.AllWebs[Utilities.GetServerRelUrlFromFullUrl(url)])
 46            {
 47                ReghostFilesInWeb(site, web, recurseWebs, force, haltOnError);
 48            }
 49            break;
 50        case "site":
 51            using (SPSite site = new SPSite(url))
 52            {
 53                ReghostFilesInSite(site, force, haltOnError);
 54            }
 55            break;
 56        case "webapplication":
 57            SPWebApplication webApp = SPWebApplication.Lookup(new Uri(url));
 58            Log("Progress: Analyzing files in web application '{0}'.", url);
 59 
 60            foreach (SPSite site in webApp.Sites)
 61            {
 62                try
 63                {
 64                    ReghostFilesInSite(site, force, haltOnError);
 65                }
 66                finally
 67                {
 68                    site.Dispose();
 69                }
 70            }
 71            break;
 72            
 73    }
 74    return OUTPUT_SUCCESS;
 75}
 76 
 77#endregion
 78 
 79/// <summary>
 80/// Reghosts the files in site.
 81/// </summary>
 82/// <param name="site">The site.</param>
 83/// <param name="force">if set to <c>true</c> [force].</param>
 84/// <param name="throwOnError">if set to <c>true</c> [throw on error].</param>
 85public static void ReghostFilesInSite(SPSite site, bool force, bool throwOnError)
 86{
 87    Log("Progress: Analyzing files in site collection '{0}'.", site.Url);
 88    foreach (SPWeb web in site.AllWebs)
 89    {
 90        try
 91        {
 92            ReghostFilesInWeb(site, web, false, force, throwOnError);
 93        }
 94        finally
 95        {
 96            web.Dispose();
 97        }
 98    }
 99}
100 
101/// <summary>
102/// Reghosts the files in web.
103/// </summary>
104/// <param name="site">The site.</param>
105/// <param name="web">The web.</param>
106/// <param name="recurseWebs">if set to <c>true</c> [recurse webs].</param>
107/// <param name="force">if set to <c>true</c> [force].</param>
108/// <param name="throwOnError">if set to <c>true</c> [throw on error].</param>
109public static void ReghostFilesInWeb(SPSite site, SPWeb web, bool recurseWebs, bool force, bool throwOnError)
110{
111    Log("Progress: Analyzing files in web '{0}'.", web.Url);
112    foreach (SPFile file in web.Files)
113    {
114        if (file.CustomizedPageStatus != SPCustomizedPageStatus.Customized && !force)
115            continue;
116 
117        Reghost(site, web, file, force, throwOnError);    
118    }
119    foreach (SPList list in web.Lists)
120    {
121        ReghostFilesInList(site, web, list, force, throwOnError);
122    }
123    
124    if (recurseWebs)
125    {
126        foreach (SPWeb childWeb in web.Webs)
127        {
128            try
129            {
130                ReghostFilesInWeb(site, childWeb, true, force, throwOnError);
131            }
132            finally
133            {
134                childWeb.Dispose();
135            }
136        }
137    }
138}
139 
140/// <summary>
141/// Reghosts the files in list.
142/// </summary>
143/// <param name="site">The site.</param>
144/// <param name="web">The web.</param>
145/// <param name="list">The list.</param>
146/// <param name="force">if set to <c>true</c> [force].</param>
147/// <param name="throwOnError">if set to <c>true</c> [throw on error].</param>
148public static void ReghostFilesInList(SPSite site, SPWeb web, SPList list, bool force, bool throwOnError)
149{
150    if (list.BaseType != SPBaseType.DocumentLibrary)
151        return;
152 
153    Log("Progress: Analyzing files in list '{0}'.", list.RootFolder.ServerRelativeUrl);
154 
155    foreach (SPListItem item in list.Items)
156    {
157        if (item.File == null)
158            continue;
159 
160        Reghost(site, web, item.File, force, throwOnError);
161    }
162}
163 
164/// <summary>
165/// Reghosts the specified file.
166/// </summary>
167/// <param name="site">The site.</param>
168/// <param name="web">The web.</param>
169/// <param name="file">The file.</param>
170/// <param name="force">if set to <c>true</c> [force].</param>
171/// <param name="throwOnError">if set to <c>true</c> [throw on error].</param>
172public static void Reghost(SPSite site, SPWeb web, SPFile file, bool force, bool throwOnError)
173{
174    try
175    {
176        string fileUrl = site.MakeFullUrl(file.ServerRelativeUrl);
177        if (file.CustomizedPageStatus != SPCustomizedPageStatus.Customized && !force)
178        {
179            Log("Progress: " + file.ServerRelativeUrl + " was not unghosted (customized).");
180            return;
181        }
182        if (file.CustomizedPageStatus != SPCustomizedPageStatus.Customized && force)
183        {
184            if (!string.IsNullOrEmpty((string)file.Properties["vti_setuppath"]))
185            {
186                file.Properties["vti_hasdefaultcontent"] = "false";
187 
188                string setupPath = (string)file.Properties["vti_setuppath"];
189                string rootPath = SPUtility.GetGenericSetupPath("Template");
190 
191                if (!File.Exists(Path.Combine(rootPath, setupPath)))
192                {
193                    string message = "The template file (" + Path.Combine(rootPath, setupPath) +
194                                     ") does not exist so re-ghosting (uncustomizing) will not be possible.";
195 
196                    // something's wrong with the setup path - lets see if we can fix it
197                    // Try and remove a leading locale if present
198                    setupPath = "SiteTemplates\\" + setupPath.Substring(5);
199                    if (File.Exists(Path.Combine(rootPath, setupPath)))
200                    {
201                        message += "  It appears that a possible template match does exist at \"" +
202                                   Path.Combine(rootPath, setupPath) +
203                                   "\" however this tool currently is not able to handle pointing the file to the correct template path.  This scenario is most likely due to an upgrade from SPS 2003.";
204 
205                        // We found a matching file so reset the property and update the file.
206                        // ---  I wish this would work but it simply doesn't - something is preventing the
207                        //      update from occuring.  Manipulating the database directly results in a 404
208                        //      when attempting to load the "fixed" page so there's gotta be something beyond
209                        //      just updating the setuppath property.
210                        //file.Properties["vti_setuppath"] = setupPath;
211                        //file.Update();
212                    }
213                    throw new FileNotFoundException(message, setupPath);
214                }
215            }
216        }
217        Log("Progress: Re-ghosting (uncustomizing) '{0}'", fileUrl);
218        file.RevertContentStream();
219 
220        file = web.GetFile(fileUrl);
221        if (file.CustomizedPageStatus == SPCustomizedPageStatus.Customized)
222        {
223            // Still unsuccessful so take measures further
224            if (force)
225            {
226                object request = Utilities.GetSPRequestObject(web);
227 
228                // I found some cases where calling this directly was the only way to force the re-ghosting of the file.
229                // I think the trick is that it's not updating the file properties after doing the revert (the
230                // RevertContentStream method will call SPRequest.UpdateFileOrFolderProperties() immediately after the 
231                // RevertContentStreams call but ommitting the update call seems to make a difference.
232                Utilities.ExecuteMethod(request, "RevertContentStreams",
233                                        new[] { typeof(string), typeof(string), typeof(bool) },
234                                        new object[] { web.Url, file.Url, file.CheckOutStatus != SPFile.SPCheckOutStatus.None });
235 
236 
237                Utilities.ExecuteMethod(file, "DirtyThisFileObject", new Type[] { }, new object[] { });
238 
239                file = web.GetFile(fileUrl);
240 
241                if (file.CustomizedPageStatus == SPCustomizedPageStatus.Customized)
242                {
243                    throw new SPException("Unable to re-ghost (uncustomize) file " + file.ServerRelativeUrl);
244                }
245                Log("Progress: " + file.ServerRelativeUrl + " was re-ghosted (uncustomized)!");
246                return;
247            }
248            throw new SPException("Unable to re-ghost (uncustomize) file " + file.ServerRelativeUrl);
249        }
250        Log("Progress: " + file.ServerRelativeUrl + " was re-ghosted (uncustomized)!");
251    }
252    catch (Exception ex)
253    {
254        if (throwOnError)
255        {
256            Log("ERROR:");
257            throw;
258        }
259        Log("ERROR: {0}", ex.Message);
260    }
261 
262}

The syntax of the command can be seen below:

C:\>stsadm -help gl-reghostfile

stsadm -o gl-reghostfile


Reghosts a file (use force to override CustomizedPageStatus check).

Parameters:
        -url <url to analyze>
        [-force]
        [-scope <WebApplication | Site | Web | List | File>]
        [-recursewebs (applies to Web scope only)]
        [-haltonerror]

The following table summarizes the command and its various parameters:

Command NameAvailabilityBuild Date
gl-reghostfileWSS v3, MOSS 2007Released: 9/13/2007, Updated: 12/14/2008
Parameter NameShort FormRequiredDescriptionExample Usage
urlYesURL to analyze. If scope is “File” then URL must point to a valid file within a Web. If scope is “List” then URL must point to a valid List within a Web.-url "http://intranet/sitedirectory/lists/sites/summary.aspx"
forceNoAttempts to force the reghosting of file(s) using internal API method calls (via reflection).-force
scopeNo (defaults to File)The scope to look at when reghosting files. Valid values are “WebApplication”, “Site”, “Web”, “List”, or “File”.-scope file
recursewebsrecurseNoApplies to “Web” scope only. If a scope of “Web” is not specified then only the single web will be considered. To recurse the web and all it’s sub-webs pass in this parameter.-recursewebs, -recurse
haltonerrorhaltNoIf an error occurs then stop processing other files within the specified scope.-haltonerror, -halt

Here’s an example of how to force the reghosting of a file:

stsadm –o gl-reghostfile -url "http://intranet/sitedirectory/lists/sites/summary.aspx" –scope file -force

If I’m able to get any answers to the issues that remain unsolved for me I’ll be sure to post them here (especially if I’m able to fix the issues).