A Better Import

Posted on Posted in SharePoint 2007, STSADM Commands

Some people may have read my post from a few days ago regarding the retargetting of the content query web part which was necessary because I had to move a sub-site from one site collection to another. In order to do the move I used the built-in import command. The problem that I sought to address with the retargetting of the content query web part was the result of the import creating new object identifiers for the lists and other such items that were imported. The only thing is that my solution only addressed the symptoms and not the actual problem and it did not address the other similar issues (such as data form web parts which suffered the same issue).

The real problem was with the import itself – internally the built-in import command sets the RetainObjectIdentity property of the SPImportSettings object to false and there is no option that you can pass into the command to override this. This is most unfortunate as this particular property has a profound affect on how objects are imported. If you set this to false then all objects are given a new ID – this is great if your intention is to duplicate a web or list within a given content database but if your intention is to move a web or list then resetting all the object IDs can really mess some things up.

For instance – if you have use the IT Team Workspace template from the Fantastic 40 application templates and attempt to move an instance of that web to another site collection via the export/import commands you will be rather annoyed to find that all of the data form web parts on the main page no longer function. If you attempt to open the web parts in SharePoint Designer to fix them you’ll find that you are unable to. The only solution is to export the web part, change the old IDs to the new IDs and then re-import the web part.

But this headache could have all been avoided if we could have told the import statement to retain the original object identifiers. So, as a solution to this problem I built a new version of the import command called gl-import2. My version functions exactly the same as the built-in version but adds to it the ability to specify whether you wish to retain the original object identifiers. If you do not wish to do this then it will call out to the built-in command thereby only using my custom version if you wish to retain object IDs.

Fortunately I already had most of the code as I had to create it for the gl-importlist command that I created. The additional code I figured out by simply using Reflector to look at what Microsoft was doing with the built-in import command. In this way I was able to make sure that my command preserved all the functionality of the original. The core code is shown below:

   1: /// <summary>
   2: /// Sets up the import object.
   3: /// </summary>
   4: /// <param name="settings">The settings.</param>
   5: /// <param name="compressFile">if set to <c>true</c> [compress file].</param>
   6: /// <param name="filename">The filename.</param>
   7: /// <param name="haltOnFatalError">if set to <c>true</c> [halt on fatal error].</param>
   8: /// <param name="haltOnWarning">if set to <c>true</c> [halt on warning].</param>
   9: /// <param name="includeusersecurity">if set to <c>true</c> [includeusersecurity].</param>
  10: /// <param name="logFile">if set to <c>true</c> [log file].</param>
  11: /// <param name="quiet">if set to <c>true</c> [quiet].</param>
  12: /// <param name="updateVersions">The update versions.</param>
  13: /// <param name="retainObjectIdentity">if set to <c>true</c> [retain object identity].</param>
  14: internal static void SetupImportObject(SPImportSettings settings, bool compressFile, string filename, bool haltOnFatalError, bool haltOnWarning, bool includeusersecurity, bool logFile, bool quiet, SPUpdateVersions updateVersions, bool retainObjectIdentity)
  15: {
  16:  settings.CommandLineVerbose = !quiet;
  17:  settings.HaltOnNonfatalError = haltOnFatalError;
  18:  settings.HaltOnWarning = haltOnWarning;
  19:  settings.FileCompression = compressFile;
  20:  
  21:  if (!compressFile)
  22:  {
  23:   if (string.IsNullOrEmpty(filename) || !Directory.Exists(filename))
  24:   {
  25:    throw new SPException(SPResource.GetString("DirectoryNotFoundExceptionMessage", new object[] { filename }));
  26:   }
  27:  }
  28:  else if (string.IsNullOrEmpty(filename) || !File.Exists(filename))
  29:  {
  30:   throw new SPException(SPResource.GetString("FileNotFoundExceptionMessage", new object[] { filename }));
  31:  }
  32:  
  33:  if (!compressFile)
  34:  {
  35:   settings.FileLocation = filename;
  36:  }
  37:  else
  38:  {
  39:   string path;
  40:   Utilities.SplitPathFile(filename, out path, out filename);
  41:   settings.FileLocation = path;
  42:   settings.BaseFileName = filename;
  43:  }
  44:  
  45:  if (logFile)
  46:  {
  47:   if (!compressFile)
  48:   {
  49:    settings.LogFilePath = Path.Combine(settings.FileLocation, "import.log");
  50:   }
  51:   else
  52:    settings.LogFilePath = Path.Combine(settings.FileLocation, filename + ".import.log");
  53:  }
  54:  
  55:  
  56:  if (includeusersecurity)
  57:  {
  58:   settings.IncludeSecurity = SPIncludeSecurity.All;
  59:   settings.UserInfoDateTime = SPImportUserInfoDateTimeOption.ImportAll;
  60:  }
  61:  settings.UpdateVersions = updateVersions;
  62:  settings.RetainObjectIdentity = retainObjectIdentity;
  63: }
  64:  
  65: /// <summary>
  66: /// Imports the site providing the option to retain the source objects ID values.  The source object
  67: /// must no longer exist in the target content database.
  68: /// </summary>
  69: /// <param name="filename">The filename.</param>
  70: /// <param name="targeturl">The targeturl.</param>
  71: /// <param name="haltOnWarning">if set to <c>true</c> [halt on warning].</param>
  72: /// <param name="haltOnFatalError">if set to <c>true</c> [halt on fatal error].</param>
  73: /// <param name="noFileCompression">if set to <c>true</c> [no file compression].</param>
  74: /// <param name="includeUserSecurity">if set to <c>true</c> [include user security].</param>
  75: /// <param name="quiet">if set to <c>true</c> [quiet].</param>
  76: /// <param name="logFile">if set to <c>true</c> [log file].</param>
  77: /// <param name="retainObjectIdentity">if set to <c>true</c> [retain object identity].</param>
  78: public void ImportSite(string filename, string targeturl, bool haltOnWarning, bool haltOnFatalError, bool noFileCompression, bool includeUserSecurity, bool quiet, bool logFile, bool retainObjectIdentity)
  79: {
  80:  if (!retainObjectIdentity)
  81:  {
  82:   // Use the built in "import" command.
  83:   ImportSite(filename, targeturl, haltOnWarning, haltOnFatalError, noFileCompression, includeUserSecurity,
  84:        quiet, logFile);
  85:   return;
  86:  }
  87:  
  88:  SPImportSettings settings = new SPImportSettings();
  89:  SPImport import = new SPImport(settings);
  90:  
  91:  SetupImportObject(settings, !noFileCompression, filename, haltOnFatalError, haltOnWarning, includeUserSecurity, logFile, quiet, SPUpdateVersions.Append, retainObjectIdentity);
  92:  
  93:  using (SPSite site = new SPSite(targeturl))
  94:  {
  95:   settings.SiteUrl = site.Url;
  96:   ExportList.ValidateUser(site);
  97:  
  98:   string dirName;
  99:   Utilities.SplitUrl(Utilities.ConvertToServiceRelUrl(Utilities.GetServerRelUrlFromFullUrl(targeturl), site.ServerRelativeUrl), out dirName, out m_webName);
 100:   m_webParentUrl = site.ServerRelativeUrl;
 101:   if (!string.IsNullOrEmpty(dirName))
 102:   {
 103:    if (!m_webParentUrl.EndsWith("/"))
 104:    {
 105:     m_webParentUrl = m_webParentUrl + "/";
 106:    }
 107:    m_webParentUrl = m_webParentUrl + dirName;
 108:   }
 109:   if (m_webName == null)
 110:   {
 111:    m_webName = string.Empty;
 112:   }
 113:  }
 114:  
 115:  EventHandler<SPDeploymentEventArgs> handler = new EventHandler<SPDeploymentEventArgs>(OnSiteImportStarted);
 116:  import.Started += handler;
 117:  
 118:  try
 119:  {
 120:   import.Run();
 121:  }
 122:  catch (SPException ex)
 123:  {
 124:   if (retainObjectIdentity && ex.Message.StartsWith("The Web site address ") && ex.Message.EndsWith(" is already in use."))
 125:   {
 126:    throw new SPException(
 127:     "You cannot import the web because the source web still exists.  Either specify the \"-deletesource\" parameter or manually delete the source web and use the exported file.", ex);
 128:  
 129:   }
 130:   else
 131:    throw;
 132:  }
 133:  finally
 134:  {
 135:   Console.WriteLine();
 136:   Console.WriteLine("Log file generated: ");
 137:   Console.WriteLine("\t{0}", settings.LogFilePath);
 138:   Console.WriteLine();
 139:  }
 140: }
 141:  
 142: /// <summary>
 143: /// Called when [site import started].
 144: /// </summary>
 145: /// <param name="sender">The sender.</param>
 146: /// <param name="args">The <see cref="Microsoft.SharePoint.Deployment.SPDeploymentEventArgs"/> instance containing the event data.</param>
 147: private void OnSiteImportStarted(object sender, SPDeploymentEventArgs args)
 148: {
 149:  SPImportObjectCollection rootObjects = args.RootObjects;
 150:  if (rootObjects.Count != 0)
 151:  {
 152:   if (rootObjects.Count != 1)
 153:   {
 154:    for (int i = 0; i < rootObjects.Count; i++)
 155:    {
 156:     if (rootObjects[i].Type == SPDeploymentObjectType.Web)
 157:     {
 158:      rootObjects[i].TargetParentUrl = m_webParentUrl;
 159:      rootObjects[i].TargetName = m_webName;
 160:      return;
 161:     }
 162:    }
 163:   }
 164:   else
 165:   {
 166:    rootObjects[0].TargetParentUrl = m_webParentUrl;
 167:    rootObjects[0].TargetName = m_webName;
 168:   }
 169:  }
 170: }

The syntax of the command I created can be seen below. With luck Microsoft will realize the benefits of this capability and will add the parameter to the built-in command with the next service pack. Note that I also added the same ability to retain object IDs to my gl-importlist command.

C:\>stsadm -help gl-import2

stsadm -o gl-import2

An improved version of the built-in "import" command.  Allows the ability to retain the original object identifiers.

Parameters:
        -url <URL to import to (parent if retainobjectidentity, otherwise full url of target location)>
        -filename <import file name>
        [-includeusersecurity]
        [-haltonwarning]
        [-haltonfatalerror]
        [-nologfile]
        [-updateversions <1-3>
            1 - Add new versions to the current file (default)
            2 - Overwrite the file and all its versions (delete then insert)
            3 - Ignore the file if it exists on the destination]
        [-nofilecompression]
        [-quiet]
        [-retainobjectidentity (the source must not exist in the content database and the url parameter must refer to the parent web)]

Here’s an example of how to import a web into a site collection while maintaining all the original object IDs:

stsadm -o gl-import2 -url "http://intranet/hr" -filename "c:\export.cmp" -includeusersecurity -retainobjectidentity

 

Update 10/11/2007: A clarification is needed on the impact of the retainobjectidentity. This parameter will affect how the URL is provided. If you set retainobjectidentity then the url must correspond to the targets parent web – a new web site will be created using the name of the exported site (so if the exported site was "http://intranet/test1" and you want the new path to be "http://teamsites/sites/site1/test1" you would specify the url as "http://teamsites/sites/site1" – note that /sites/site1/test1 cannot already exist – don’t use createweb). If you do not specify retainobjectidentity then the url parameter must be the target url (this allows you to specify a different web name) but the target web must already exist with a matching template (use createweb to create the placeholder web). If you wanted to do the same as described above but without retaining the object IDs then you would specify the url as "http://teamsites/sites/site1/test1". Also – I discovered that the retainobjectidentity switch introduces another limitation – it really only works with webs that are not sub-webs of another web (so first level child objects of a site collection). This is a limitation of the deployment API and not my command.

26 thoughts on “A Better Import

  1. SUPERB!! – This solved a massive problem that I’ve had with moving the application templates. MS would do well to follow your lead my friend.

    Thank you so much for sharing your fix.

  2. Gary,

    Have you encountered any problems with imports when versioning is enabled on the site? I have had some hit and miss problems with documents that have multiple versions. It generates a version not found error. I have not been able to pin point the problem. The command I am using is:
    stsadm.Exe -o import -url http://spqteams/moss2007/ -filename E:\Backup\moss2007.bak -includeusersecurity -updateversions 2

    Thanks,
    Ryan Johnson

  3. Unfortunately there are just tons of issues with the built-in deployment API – you may want to search through the MSDN forums for I believe I’ve seen this specific issue discussed (can’t remember if there was a solution though).

  4. I am slightly confused by your update on this post. You say it doesn’t work with subwebs? Does this mean I can only really use this to move an entire site collection around? Can I export http://localhost/myweb to another server and the same target url?

  5. Hi Gary,

    Thanks for the solution!

    I think I might be doing something wrong, but when I attempt to move my site with retainobjectidentity set, the operation shows successful but no new site was created.

    Any possible cause?

  6. Eric – that specific problem is actually why I added the retainobjectidentity parameter – if you have list view web parts, for example, that reference a specific list those web parts will be broken unless you retain the identity of the list (the list view web part references the list by it’s GUID and not it’s name or path). There are other web parts that have this same limitation.

  7. Gary,

    Thanks alot for your help! The problem has been resolved. It seems that the problem lies with exporting the site using Designer 2007.

    Anyway, your efforts are a great help to the community!

  8. Thanks for the tool. I am getting an error when runing import with the -retainobjectidentity switch. It says that the name already exists. I am importing to a new site collection. Thanks

  9. Gary,

    You’re a genius, thank you very much for this…you’ve scored me some serious brownite points!

    Thanks again.

    Simon Baynes.

  10. Hi Gary,

    I am a newbie to Sharepoint. I do not have any issues at present, but am trying to understand how I would actually use your code above to add the gl-import2 operation to stsadm.
    Thank you.

    Orville

  11. I’m getting various access denied messages along the lines of:

    Debug: Security check failed in OnListItemImport
    Warning: Access denied.

    when trying to run this command, even though I am using the farm admin account.

    Any idea’s what to look for?

    all the best

  12. Hi Gary

    When using the stsadm -o gl-import2 command I am getting an error relating to a list already existing, the import then seems to fail. Should it not handle this or am I missing something?

    I then used the UI to delete the existing list and ran the import again. It then seemed to fail at the same point saying the list exists. In the the UI the list appears but when I click it, reports that the list does not exist.

    Any pointers?

  13. Hi All,
    I am hoping that someone will read this who has done what I am currently trying to do and am experiencing issues with. I am exporting a site from a WSS 3 SP2 server into a MOSS 2007 SP2 server. The site has child/sub-sites and several meeting and document workspaces. When I imported I received an error regarding each workspace imported. Here is the error from the import log, “[4/1/2010 2:01:22 AM]: Progress: Importing ListItem /teams/pm/Partnerships/WorldView/_catalogs/masterpage?id=1.
    [4/1/2010 2:01:22 AM]: Verbose: Deleting…
    [4/1/2010 2:01:22 AM]: Warning: File cannot be deleted will try to append the file instead.
    *** Inner exception:
    Cannot remove file “default.master”. Error Code: 158.
    at Microsoft.SharePoint.Library.SPRequest.AddOrDeleteUrl(String bstrUrl, String bstrDirName, Boolean bAdd, UInt32 dwDeleteOp, Int32 iUserId, Guid& pgDeleteTransactionId)
    at Microsoft.SharePoint.Deployment.FileSerializer.DeleteFileForOverride(Object fileOrListItem, SPWeb web, String fileUrl, SPImportSettings settings)
    [4/1/2010 2:01:22 AM]: Progress: Importing ListItem /teams/pm/Partnerships/WorldView/_catalogs/masterpage?id=2.”

    I have the updated STSADM extensions installed on both systems as well. Any help you can provide will be greatly appreciated!

  14. Hi All,

    A little old, but worth giving it a try. The issues expressed above are exactly the symptoms I have in migrating a site set to another, (original based in SPS2003 and new based in MOSS2007). It appears that the coding for this utility has been removed. Whilst I am running with the latest MOSS SP2/updates, still no joy. Getting desperate. Any chance of having the code replaced :).

    Thanks in advance,

    Cheers,

    Phil Showell.

    1. Looks like the post got only partially migrated when I moved from my old blog. The full post is there now – that said, the command was always available via the WSP download.

  15. This is brilliant and exactly what I needed.

    The User Information List is imported but despite using -retainobjectidentity it still generates a new GUID.

    Any ideas how I can overcome this as I have a dataview for this list on several pages which are now broken. Any changes I make to these pages cannot be promoted to the next site without having to update the GUID.

    Ta!!

    1. The user information list is a bit of a special list so I’d honestly be surprised if it did work with it (though I’ve never attempted to migrate that list so really not sure).

  16. I’ve used the gl-import2 command using the retainobjectidentity switch to import a whole site collection. Now on sub-sites of this SC, if I go to Site Actions/Create, I only see the Web Pages column (I don’t see columns to create Document Libraries or Lists). Any ideas?
    (I’m running WSS 3.0 on 2003 servers with SQL 2005. I used the stsadm -o export for exporting the site collection).

    1. I found the Team Collaboaration Lists feature was disabled on all sub sites after the import. Just had to Activate on each site to get this working again. Not sure why this happened…

  17. Thanks for your extensions! I’ve seen both the default import and also your improved import working best when executed twice. With the first import all lists and libraries are created and data is imported and the second import are all web parts created that are dependend upon lists or libraries. Your import operation is also better whith this approach because the default import operation inserts all content twice (probably because it doesn’t retain the object identity).

    One problem we are still facing is the following:
    Cannot import site. The exported site is using language 1043 but the destination site is using language 1033. You can import sites only into sites that are using same language as the exported site.

    This error occurs when the site collection is of a different language while the (sub)site on which the import takes place is correct (1043). The default import operation does not give this error!

    A simple work around is deleting the site collection and creating a new one with the correct language, but this isn’t possible in all situations because some site collections already contain other sites. Hope you can help me out.

Leave a Reply

Your email address will not be published. Required fields are marked *

*