SharePoint Automation Gary Lapointe – Founding Partner, Aptillon, Inc.

18May/085

Setting Metadata

In my last post I mentioned a project which required me to move documents from one list to another list in a different farm one folder at a time.  Along with that was a requirement to set various field values (metadata) based on patterns in the folder name and/or filename.  I needed a reasonably flexible way to accomplish this considering that the client didn't actually have a clue as to what they really wanted the rules to be.  I already had a command (gl-replacefieldvalues) which let me set the value of an existing field but it didn't allow me to do it based on the values of other fields and there was not real filtering capability.  So I built a new command called gl-setmetadata which allows me to pass in an XML file containing various rules.

There's really not much to the code - the bulk of it is just parsing the XML and figuring out what to do.  There's two core methods - the first, ProcessFolder, is responsible for getting the collection of items that should be processed using the provided rules.  This is done by using an SPQuery object and passing in the Query XML node if present.  The second method, ApplyRule, is called by the ProcessFolder method for each Rule node found in the XML and it is responsible for setting any field data based on the rules.

   1: public SetMetaData()
   2: {
   3:     SPParamCollection parameters = new SPParamCollection();
   4:     parameters.Add(new SPParam("url", "url", true, null, new SPNonEmptyValidator(), "Please specify the url to search."));
   5:     parameters.Add(new SPParam("quiet", "q"));
   6:     parameters.Add(new SPParam("test", "t"));
   7:     parameters.Add(new SPParam("inputfile", "input", true, null, new SPFileExistsValidator()));
   8:     parameters.Add(new SPParam("logfile", "log", false, null, new SPDirectoryExistsAndValidFileNameValidator()));
   9:     parameters.Add(new SPParam("recursefolders", "recurse"));
  10:  
  11:     StringBuilder sb = new StringBuilder();
  12:     sb.Append("\r\n\r\nUpdates list field values based on the rules defined in the provided input file.  Use -test to verify your updates before executing.\r\n\r\nParameters:");
  13:     sb.Append("\r\n\t-url <list folder url>");
  14:     sb.Append("\r\n\t-inputfile <input file containing meta data rules>");
  15:     sb.Append("\r\n\t[-recursefolders]");
  16:     sb.Append("\r\n\t[-quiet]");
  17:     sb.Append("\r\n\t[-test]");
  18:     sb.Append("\r\n\t[-logfile <log file>]");
  19:  
  20:     Init(parameters, sb.ToString());
  21: }
  22:  
  23: /// <summary>
  24: /// Gets the help message.
  25: /// </summary>
  26: /// <param name="command">The command.</param>
  27: /// <returns></returns>
  28: public override string GetHelpMessage(string command)
  29: {
  30:     return HelpMessage;
  31: }
  32:  
  33: /// <summary>
  34: /// Runs the specified command.
  35: /// </summary>
  36: /// <param name="command">The command.</param>
  37: /// <param name="keyValues">The key values.</param>
  38: /// <param name="output">The output.</param>
  39: /// <returns></returns>
  40: public override int Execute(string command, StringDictionary keyValues, out string output)
  41: {
  42:     output = string.Empty;            
  43:  
  44:     string url = Params["url"].Value.TrimEnd('/');
  45:     bool quiet = Params["quiet"].UserTypedIn;
  46:     bool testMode = Params["test"].UserTypedIn;
  47:     string logFile = Params["logfile"].Value;
  48:     XmlDocument metaDataDoc = new XmlDocument();
  49:     string inputFile = Params["inputfile"].Value;
  50:     bool recurseFolders = Params["recursefolders"].UserTypedIn;
  51:  
  52:     Verbose = !quiet;
  53:     LogFile = logFile;
  54:  
  55:     metaDataDoc.Load(inputFile);
  56:  
  57:     using (SPSite site = new SPSite(url))
  58:     using (SPWeb web = site.OpenWeb())
  59:     {
  60:         SPFolder folder = web.GetFolder(url);
  61:  
  62:         if (!folder.Exists || folder == null) // the null check is unnecessary but it makes me feel better.
  63:             throw new SPException("The specified list folder was not found.");
  64:  
  65:         SPList list = null;
  66:         try
  67:         {
  68:             list = web.Lists[folder.ParentListId];
  69:         }
  70:         catch (ArgumentException)
  71:         {}
  72:         if (list == null) // This should never happen if we found a folder but again, it makes me feel better having it.
  73:             throw new SPException("The specified list was not found.");
  74:  
  75:         // Process the folder.
  76:         ProcessFolder(folder, list, metaDataDoc, recurseFolders, testMode);
  77:     }
  78:     return OUTPUT_SUCCESS;
  79: }
  80:  
  81: /// <summary>
  82: /// Processes the folder.
  83: /// </summary>
  84: /// <param name="folder">The folder.</param>
  85: /// <param name="list">The list.</param>
  86: /// <param name="metaDataDoc">The meta data doc.</param>
  87: /// <param name="recurseFolders">if set to <c>true</c> [recurse folders].</param>
  88: /// <param name="testMode">if set to <c>true</c> [test mode].</param>
  89: private static void ProcessFolder(SPFolder folder, SPList list, XmlDocument metaDataDoc, bool recurseFolders, bool testMode)
  90: {
  91:     // If we don't have any rules to process then there's no sense continueing so error out.
  92:     if (metaDataDoc.SelectNodes("//Rule").Count == 0)
  93:         throw new SPException("Missing \"Rule\" node(s) which should be a child of the root \"MetaData\" node.");
  94:  
  95:     // Get a namespace manager so that we can retrieve the Query element if present.
  96:     XmlNamespaceManager nsManager = new XmlNamespaceManager(metaDataDoc.NameTable);
  97:     nsManager.AddNamespace("sp", "http://schemas.microsoft.com/sharepoint/");
  98:  
  99:     // Look for a Query element
 100:     XmlElement queryElement = (XmlElement)metaDataDoc.SelectSingleNode("//sp:Query", nsManager);
 101:     SPListItemCollection items;
 102:     SPQuery query = new SPQuery();
 103:     if (recurseFolders)
 104:         query.ViewAttributes = "Scope=\"Recursive\"";
 105:     // Set the root folder to query
 106:     query.Folder = folder;
 107:     if (queryElement != null)
 108:     {
 109:         // We have a query element so do an intial filtering using the provided filter
 110:         query.Query = queryElement.OuterXml;
 111:         items = list.GetItems(query);
 112:     }
 113:     else
 114:     {
 115:         // User didn't provide any query parameters so just use an empty query (no filtering)
 116:         items = list.GetItems(query);
 117:     }
 118:  
 119:     Log("Beginning processing of {0} items...", items.Count.ToString());
 120:     int modificationCount = 0;
 121:  
 122:     for (int i = 0; i < items.Count; i++)
 123:     {
 124:         SPListItem item = items[i];
 125:         Log("Progress: Processing item {0}: {1}\r\n", item.ID.ToString(), item["ServerUrl"].ToString());
 126:  
 127:         if (item.FileSystemObjectType == SPFileSystemObjectType.Folder)
 128:         {
 129:             // Currently not handling folders - no particular reason, I just don't need this ability.
 130:             // Commenting out this block will not hurt anything.
 131:             Log("Progress: Item {0} is a folder - skipping.", item.ID.ToString());
 132:             continue;
 133:         }
 134:  
 135:         bool modified = false;
 136:  
 137:         // Loop through each rule element and apply the rules changes
 138:         foreach (XmlElement ruleElement in metaDataDoc.SelectNodes("//Rule"))
 139:         {
 140:             if (ApplyRule(item, ruleElement))
 141:                 modified = true;
 142:         }
 143:  
 144:         if (modified)
 145:         {
 146:             // The rules resulted in modified data so update the item if not in test mode.
 147:             if (!testMode)
 148:                 item.SystemUpdate();
 149:             modificationCount++;
 150:             Log("Progress: Item ID {0} was modified.", item.ID.ToString());
 151:         }
 152:         else
 153:         {
 154:             // There were no modifications made
 155:             Log("Progress: Item ID {0} was NOT modified.", item.ID.ToString());
 156:         }
 157:  
 158:         Log("Progress: Finished Processing item {0}\r\n\r\n", item.ID.ToString());
 159:  
 160:     }
 161:     Log("Finished processing items.  {0} out of {1} items were modified.\r\n", modificationCount.ToString(), items.Count.ToString());
 162:  
 163: }
 164:  
 165: /// <summary>
 166: /// Applies the rule.
 167: /// </summary>
 168: /// <param name="item">The item.</param>
 169: /// <param name="ruleElement">The rule element.</param>
 170: /// <returns></returns>
 171: private static bool ApplyRule(SPListItem item, XmlElement ruleElement)
 172: {
 173:     bool modified = false;
 174:     string ruleName = ruleElement.GetAttribute("Name");
 175:  
 176:     XmlElement matchElement = (XmlElement)ruleElement.SelectSingleNode("Match");
 177:     bool isMatch = true;
 178:             
 179:     // The match element is optional and just provides some additional regular expression filtering beyond what the Query element can provide
 180:     if (matchElement != null)
 181:     {
 182:         bool isAnd = true;
 183:         if (matchElement.HasAttribute("Op"))
 184:             isAnd = matchElement.GetAttribute("Op").ToLowerInvariant() == "and";
 185:         // For "And" operations we default our starter item to true as everything must come back as true to be a match
 186:         // For "Or" operations we default our starter item to false as we only need one item to come back as true to 
 187:         // be a match and we don't want that one item to be the starter item.
 188:         bool fieldMatches = isAnd;
 189:                 
 190:         // If we have a Match element then we need at least one Field element otherwise what's the point.
 191:         if (matchElement.SelectNodes("Field").Count == 0)
 192:             throw new SPException("Missing \"Field\" node(s) which should be a child of the \"Match\" node.");
 193:  
 194:         foreach (XmlElement fieldElement in matchElement.SelectNodes("Field"))
 195:         {
 196:             // The Field element needs a Name attribute and a value to use as the search pattern string
 197:             if (!fieldElement.HasAttribute("Name"))
 198:                 throw new SPException("Missing \"Name\" attribute of \"Field\" node.");
 199:             if (string.IsNullOrEmpty(fieldElement.InnerText.Trim()))
 200:                 throw new SPException(string.Format("Missing search pattern string value for match field '{0}'", fieldElement.GetAttribute("Name")));
 201:  
 202:             // We use the internal name for all field names
 203:             SPField field = item.Fields.GetFieldByInternalName(fieldElement.GetAttribute("Name"));
 204:  
 205:             // Determine if we have a match for this field.
 206:             bool fieldMatch = Regex.IsMatch(item[field.Id].ToString(), fieldElement.InnerText);
 207:  
 208:             // Apply the match results to our fieldMatches variable to track the overall result
 209:             if (isAnd)
 210:                 fieldMatches = fieldMatches && fieldMatch;
 211:             else
 212:                 fieldMatches = fieldMatches || fieldMatch;
 213:         }
 214:         // Set the overall result
 215:         isMatch = fieldMatches;
 216:     }
 217:     if (!isMatch)
 218:     {
 219:         Log("Progress: Unable to find match for rule '{0}'.", ruleName);
 220:         return modified; // No match so evaluate the next rule
 221:     }
 222:     else
 223:         Log("Progress: Found match for rule '{0}'.", ruleName);
 224:  
 225:     // Every Rule element must have one and only one Set element
 226:     XmlElement setElement = (XmlElement) ruleElement.SelectSingleNode("Set");
 227:     if (setElement == null)
 228:         throw new SPException("Missing \"Set\" node.");
 229:  
 230:     // Every Set element must have at least one Field element
 231:     if (setElement.SelectNodes("Field").Count == 0)
 232:         throw new SPException("Missing \"Field\" node(s) which should be a child of the \"Set\" node.");
 233:  
 234:     // Loop through all the Field elements and apply the indicated values
 235:     foreach (XmlElement fieldElement in setElement.SelectNodes("Field"))
 236:     {
 237:         // Every Field element must have a Name attribute - the value can be empty which is the same as setting the field to null.
 238:         if (!fieldElement.HasAttribute("Name"))
 239:             throw new SPException("Missing \"Name\" attribute of \"Field\" node.");
 240:  
 241:         string fieldName = fieldElement.GetAttribute("Name");
 242:         string fieldData = fieldElement.InnerText;
 243:         SPField field = item.Fields.GetFieldByInternalName(fieldName);
 244:  
 245:         if (field.ReadOnlyField)
 246:         {
 247:             // We can't update read-only fields so log a warning and move on.
 248:             Log("WARNING: Field '{0}' is read only and will not be updated.", EventLogEntryType.Warning, field.InternalName);
 249:             continue;
 250:         }
 251:  
 252:         if (field.Type == SPFieldType.Computed)
 253:         {
 254:             // We can't update computed fields so log a warning and move on.
 255:             Log("Progress: Field '{0}' is a computed column and will not be updated.", EventLogEntryType.Warning, field.InternalName);
 256:             continue;
 257:         }
 258:         // If a SearchPattern attribute was provided then do a regular expression replace instead of just a straight up set.
 259:         if (fieldElement.HasAttribute("SearchPattern"))
 260:         {
 261:             if (string.IsNullOrEmpty(fieldElement.GetAttribute("SearchPattern")))
 262:                 throw new SPException(string.Format("SearchPattern attribute of Field node '{0}' is empty.", fieldName));
 263:             
 264:             if (item[field.Id] == null)
 265:             {
 266:                 // We can't do a regex on a null value so move on
 267:                 Log("Progress: Value of field '{0}' is 'null' - no replace operation will be performed.", field.InternalName);
 268:                 continue;
 269:             }
 270:             else
 271:                 fieldData = Regex.Replace(item[field.Id].ToString(), fieldElement.GetAttribute("SearchPattern"), fieldData);
 272:         }
 273:         // If the fieldData is empty then make sure it's set to null
 274:         if (string.IsNullOrEmpty(fieldData))
 275:             fieldData = null;
 276:  
 277:         
 278:         if (item[field.Id] == null || item[field.Id].ToString() != fieldData)
 279:         {
 280:             // The modified field data is different from the source so go ahead and apply the change
 281:             Log("Progress: Applying modification to field '{0}' per rule '{1}'", fieldName, ruleName);
 282:             if (field.Type == SPFieldType.URL)
 283:                 item[field.Id] = new SPFieldUrlValue(fieldData);
 284:             else
 285:                 item[field.Id] = fieldData;
 286:  
 287:             modified = true;
 288:         }
 289:         else
 290:         {
 291:             Log("Progress: No change required for field '{0}' per rule '{1}'.", fieldName, ruleName);
 292:         }
 293:     }
 294:     if (!modified)
 295:         Log("Progress: Set rules resulted in no change from existing data for rule '{0}'.", ruleName);
 296:  
 297:     return modified;
 298: }

The core thing to understand with this command is the structure of the input folder and this where things get a little more complicated.  I don't currently have an XSD for this (I may create one to aid in validation but I just didn't have the time).  So failing a good XSD here's a reasonably detailed example XML file with comments:

   1: <MetaData>
   2:     <!-- Query is an optional CAML element and is used to filter the items that are to be considered.  Anything you can do with a standard CAML Query element you can put here (be sure to include the namespace attribute) -->
   3:     <Query xmlns="http://schemas.microsoft.com/sharepoint/">
   4:         <Where>
   5:             <BeginsWith>
   6:                 <FieldRef Name="FileRef" />
   7:                 <Value Type="string">/Documents/Sub-Folder1/</Value>
   8:             </BeginsWith>
   9:         </Where>
  10:     </Query>
  11:     <!-- There must be at least one Rule element - multiple elements are processed in the order they appear -->
  12:     <!-- The Rule element may contain an optional Name attribute which is a simple label used for logging -->
  13:     <Rule Name="Set Content Type">
  14:         <!-- Every Rule element must have one and only one Set element -->
  15:         <Set>
  16:             <!-- The Set element must contain one or more Field elements -->
  17:             <!-- The Field element must have a Name attribute which corresponds to the fields internal name -->
  18:             <!-- The value of the Field element is what will be set to the list item for that field -->
  19:             <!-- A Field element may contain an optional SearchPattern attribute which can be used to update an existing value via a Regex.Replace() call -->
  20:             <!-- If no SearchPattern attribute is present then existing data is ignored -->
  21:             <Field Name="ContentType">Dublin Core Columns</Field>
  22:         </Set>
  23:     </Rule>
  24:     <Rule Name="Set English Language">
  25:         <!-- A Rule element can contain one optional Match element which is used to provide regular expression based filtering -->
  26:         <!-- The Match element can contain an optional Op attribute used to indicate whether the match logic is "AND" or "OR" (default is "AND" if not present) -->
  27:         <Match Op="OR">
  28:             <!-- The Field element must have a Name attribute which corresponds to the fields internal name -->
  29:             <!-- The value of the Field element is used in a Regex.IsMatch() call to determine whether the item should be processed -->
  30:             <Field Name="FileLeafRef">(?i:.* Eng.*|.*ENGLISH ONLY.*|.*-EN.*)</Field>
  31:             <Field Name="Title">(?i:.* Eng.*|.*ENGLISH ONLY.*|.*-EN.*)</Field>
  32:         </Match>
  33:         <Set>
  34:             <Field Name="FileLeafRef" SearchPattern="(?i: -?Eng|ENGLISH ONLY)|-EN">-English</Field>
  35:             <Field Name="Language">English</Field>
  36:         </Set>
  37:     </Rule>
  38:     <Rule Name="Set Korean Language">
  39:         <Match Op="And">
  40:             <Field Name="FileLeafRef">(?i:.* Kor.*|.*KOREAN ONLY.*|.*-KO.*)</Field>
  41:         </Match>
  42:         <Set>
  43:             <Field Name="FileLeafRef" SearchPattern="(?i: -?Kor|KOREAN ONLY)|-KO">-Korean</Field>
  44:             <Field Name="Language">Korean</Field>
  45:         </Set>
  46:     </Rule>
  47: </MetaData>

Note that I don't claim to be a regular expression expert and I've not extensively tested the regular expressions in the examples above and I know that there are issues with them for more complex data but for the purpose of a simple demonstration they do well enough.  The example above will return back all documents in the folder "/documents/sub-folder1" and will set the content type of every item to "Dublin Core Columns".  It will then standardize the name of the file (FileLeafRef) so that it only contains "*-English" or "*-Korean" using information in the filename and it will also set the Language field to English or Korean using this same information.

Probably the most important thing to remember when constructing your XML is that you need to use the internal field name and not the display name.

You can also do additional filtering using the command line parameters by restricting whether folders are recursed and by specifying a sub-folder instead of a root list folder.  The syntax of the command can be seen below:

C:\>stsadm -help gl-setmetadata stsadm -o gl-setmetadata Updates list field values based on the rules defined in the provided input file. Use -test to verify your updates before executing. Parameters: -url <list folder url> -inputfile <input file containing meta data rules> [-recursefolders] [-quiet] [-test] [-logfile <log file>]

Here's an example of how you would execute this command using the XML shown above as an input:

stsadm -o gl-setmetadata -url http://portal/documents -inputfile c:\metadata.xml -recursefolders -logfile c:\metadata.log

Like many of my commands that do batch updating you can run this command in a test mode by passing in a "-test" parameter.

18Feb/0817

Import Site Columns

In my last post I wrote about a command that I created to export site columns to an xml file because I needed the data for a Feature I was working on. But what if you just want to create a column using the exported results and for whatever reason you don't want to create a Feature? I figured it would take me about 2 minutes to throw together an import command that would take in the results of the export so may as well add it to the collection. So now you can use the gl-importsitecolumns command to import what you exported using gl-exportsitecolumns (useful if you're just trying to move some fields around though I'd still recommend you do this via a Feature). Note that this command will blow up if a column already exists with the same name. The code for this is really simple - just loop through the XML and call the SPFieldCollection's AddFieldAsXml method: 

   1:  string url = Params["url"].Value;
   2:   
   3:  XmlDocument xmlDoc = new XmlDocument();
   4:  xmlDoc.Load(Params["inputfile"].Value);
   5:   
   6:   
   7:  // Get the source content type and fields.
   8:  using (SPSite site = new SPSite(url))
   9:  using (SPWeb web = site.AllWebs[Utilities.GetServerRelUrlFromFullUrl(url)])
  10:  {
  11:      foreach (XmlElement fieldNode in xmlDoc.SelectNodes("//Field"))
  12:      {
  13:          web.Fields.AddFieldAsXml(fieldNode.OuterXml);
  14:      }
  15:  }
The syntax of the command can be seen below:
C:\>stsadm -help gl-importsitecolumns

stsadm -o gl-importsitecolumns

Imports one or more site fields (columns) to a site.

Parameters:
        -url <url>
        -inputfile <file to import fields from>
Here's an example of how to import site columns that were exported using the exportsitecolumns command:
stsadm -o gl-importsitecolumns -url "http://intranet" -inputfile c:\fields.xml

17Feb/089

Export Site Columns

As I mentioned in my previous post about the gl-exportcontenttypes command I need to be able to quickly and easily get the CAML necessary to recreate site columns in a Feature. To do this I created a quick and dirty command called gl-exportsitecolumns. The code for this is extremely simple - I just get the SPField objects of interest based on the parameters passed in and dump out the SchemaXml property - that's it:

   1:  string url = Params["url"].Value;
   2:  string fieldTitle = Params["fielddisplayname"].Value;
   3:  string fieldName = Params["fieldinternalname"].Value;
   4:  string groupName = Params["group"].Value;
   5:  bool useTitle = Params["fielddisplayname"].UserTypedIn;
   6:  bool useName = Params["fieldinternalname"].UserTypedIn;
   7:  bool useGroup = Params["group"].UserTypedIn;
   8:  bool all = !(useTitle || useName || useGroup);
   9:   
  10:  StringBuilder sb = new StringBuilder();
  11:  XmlTextWriter xmlWriter = new XmlTextWriter(new StringWriter(sb));
  12:  xmlWriter.Formatting = Formatting.Indented;
  13:   
  14:  try
  15:  {
  16:   xmlWriter.WriteStartElement("Elements");
  17:   
  18:   // Get the source content type and fields.
  19:   using (SPSite site = new SPSite(url))
  20:   {
  21:    using (SPWeb web = site.AllWebs[Utilities.GetServerRelUrlFromFullUrl(url)])
  22:    {
  23:     SPFieldCollection fields = web.Fields;
  24:     foreach (SPField field in fields)
  25:     {
  26:      if (all ||
  27:       (useGroup && groupName.ToLowerInvariant() == field.Group.ToLowerInvariant()) ||
  28:       (useName && fieldName.ToLowerInvariant() == field.InternalName.ToLowerInvariant()) ||
  29:       (useTitle && fieldTitle.ToLowerInvariant() == field.Title.ToLowerInvariant()))
  30:      {
  31:       xmlWriter.WriteString("\r\n");
  32:       xmlWriter.WriteRaw(field.SchemaXml);
  33:      }
  34:     }
  35:    }
  36:   }
  37:   
  38:   xmlWriter.WriteString("\r\n");
  39:   xmlWriter.WriteEndElement(); // Elements
  40:  }
  41:  finally
  42:  {
  43:   xmlWriter.Flush();
  44:   xmlWriter.Close();
  45:  }
  46:  File.WriteAllText(Params["outputfile"].Value, sb.ToString());
The syntax of the command can be seen below:
C:\>stsadm -help gl-exportsitecolumns

stsadm -o gl-exportsitecolumns


Exports one or more site fields (columns) to a file.

Parameters:
        -url <url>
        {[-fielddisplayname <field display name> / -fieldinternalname <field internal name>]
         [-group <site column group name to filter results by>]}
        -outputfile <file to output field schema to>
Here's an example of how to dump the XML for all the site columns in a particular group:
stsadm -o gl-exportsitecolumns -url "http://intranet" -outputfile c:\fields.xml -group "Custom Site Columns"

17Feb/0822

Export Content Types

I was recently working on a client project where there was an environment already setup with several custom content types and custom site columns that had been created. We wanted to reverse engineer those content types and columns so that we could create a Feature that could be used to deploy them into the various environments. I poked around a bit and couldn't find anything that would simply give me the CAML that I needed for my Feature. So I decided to create a couple of new commands to help me do this quickly: gl-exportcontenttypes and gl-exportsitecolumns.

I'll cover gl-exportsitecolumns in a follow up post. Looking at the code you can see that there's really not much going on. I simply grab the content type(s) to export and get the SchemaXml property. One thing of note is that the SchemaXml property returns back XML that contains the complete field definition - because this won't work in a Feature I replace the <Field /> elements with a corresponding <FieldRef /> element. If the field definitions are needed then you can simply provide the includefielddefinitions parameter. Note that I'm outputting everything in one file but if you intend to use this in a Feature you'll want to break this up into multiple files. For the field definitions I also just output the SchemaXml of the SPField objects associated with the content type (note that there will be some attributes that you'll need to pull when using the resultant output in a Feature). To get the list bindings I loop through all the lists throughout the site collection and add a ContentTypeBindings element for each list that contains a reference to the identified content types.

   1: public class ExportContentTypes : SPOperation
   2: {
   3:     private const string ENCODED_SPACE = "_x0020_";
   4:     /// <summary>
   5:     /// Initializes a new instance of the <see cref="CopyContentTypes"/> class.
   6:     /// </summary>
   7:     public ExportContentTypes()
   8:     {
   9:         SPParamCollection parameters = new SPParamCollection();
  10:         parameters.Add(new SPParam("url", "url", true, null, new SPUrlValidator(), "Please specify the source site collection."));
  11:         parameters.Add(new SPParam("name", "n", false, null, new SPNonEmptyValidator()));
  12:         parameters.Add(new SPParam("group", "g", false, null, new SPNonEmptyValidator()));
  13:         parameters.Add(new SPParam("listname", "list", false, null, new SPNonEmptyValidator()));
  14:         parameters.Add(new SPParam("outputfile", "output", false, null, new SPDirectoryExistsAndValidFileNameValidator()));
  15:         parameters.Add(new SPParam("includelistbindings", "ilb"));
  16:         parameters.Add(new SPParam("includefielddefinitions", "ifd"));
  17:         parameters.Add(new SPParam("excludeparentfields", "epf"));
  18:         parameters.Add(new SPParam("removeencodedspaces", "res"));
  19:  
  20:         StringBuilder sb = new StringBuilder();
  21:         sb.Append("\r\n\r\nExports Content Types to an XML file.\r\n\r\nParameters:");
  22:         sb.Append("\r\n\t-url <url containing the content types>");
  23:         sb.Append("\r\n\t-outputfile <file to output results to>");
  24:         sb.Append("\r\n\t[-name <name of an individual content type to export>]");
  25:         sb.Append("\r\n\t[-group <content type group name to filter results by>]");
  26:         sb.Append("\r\n\t[-listname <name of a list to export content types from>]");
  27:         sb.Append("\r\n\t[-includelistbindings]");
  28:         sb.Append("\r\n\t[-includefielddefinitions]");
  29:         sb.Append("\r\n\t[-excludeparentfields]");
  30:         sb.Append("\r\n\t[-removeencodedspaces (removes '_x0020_' in field names)]");
  31:         Init(parameters, sb.ToString());
  32:     }
  33:  
  34:     #region ISPStsadmCommand Members
  35:  
  36:     /// <summary>
  37:     /// Gets the help message.
  38:     /// </summary>
  39:     /// <param name="command">The command.</param>
  40:     /// <returns></returns>
  41:     public override string GetHelpMessage(string command)
  42:     {
  43:         return HelpMessage;
  44:     }
  45:  
  46:     /// <summary>
  47:     /// Runs the specified command.
  48:     /// </summary>
  49:     /// <param name="command">The command.</param>
  50:     /// <param name="keyValues">The key values.</param>
  51:     /// <param name="output">The output.</param>
  52:     /// <returns></returns>
  53:     public override int Run(string command, StringDictionary keyValues, out string output)
  54:     {
  55:         output = string.Empty;
  56:  
  57:         InitParameters(keyValues);
  58:  
  59:         string url = Params["url"].Value.TrimEnd('/');
  60:         string contentTypeName = null;
  61:         if (Params["name"].UserTypedIn)
  62:             contentTypeName = Params["name"].Value;
  63:         string contentTypeGroup = null;
  64:         if (Params["group"].UserTypedIn)
  65:             contentTypeGroup = Params["group"].Value.ToLowerInvariant();
  66:         if (Params["group"].UserTypedIn && Params["listname"].UserTypedIn)
  67:             throw new SPSyntaxException("The parameters group and listname are incompatible");
  68:         bool excludeParentFields = Params["excludeparentfields"].UserTypedIn;
  69:         bool removeEncodedSpaces = Params["removeencodedspaces"].UserTypedIn;
  70:  
  71:         StringBuilder sb = new StringBuilder();
  72:         XmlTextWriter xmlWriter = new XmlTextWriter(new StringWriter(sb));
  73:         xmlWriter.Formatting = Formatting.Indented;
  74:  
  75:         Dictionary<Guid, SPField> ctFields = new Dictionary<Guid, SPField>();
  76:  
  77:         using (SPSite site = new SPSite(url))
  78:         {
  79:             using (SPWeb web = site.AllWebs[Utilities.GetServerRelUrlFromFullUrl(url)])
  80:             {
  81:                 SPContentTypeCollection availableContentTypes;
  82:  
  83:                 if (Params["listname"].UserTypedIn)
  84:                 {
  85:                     SPList list = web.Lists[Params["listname"].Value];
  86:                     availableContentTypes = list.ContentTypes;
  87:                 }
  88:                 else
  89:                 {
  90:                     availableContentTypes = web.AvailableContentTypes;
  91:                 }
  92:                 List<SPContentType> contentTypes = new List<SPContentType>();
  93:                 try
  94:                 {
  95:                     xmlWriter.WriteStartElement("Elements");
  96:                     xmlWriter.WriteAttributeString("xmlns", "http://schemas.microsoft.com/sharepoint/");
  97:  
  98:                     // Gather up all the content types we want to export out.
  99:                     if (contentTypeName != null)
 100:                     {
 101:                         SPContentType ct = availableContentTypes[contentTypeName];
 102:                         if (ct == null)
 103:                         {
 104:                             output += "The content type specified could not be found.";
 105:                             return 0;
 106:                         }
 107:                         else
 108:                         {
 109:                             contentTypes.Add(ct);
 110:                         }
 111:                     }
 112:                     else
 113:                     {
 114:                         // Loop through all the source content types and create them at the target.
 115:                         foreach (SPContentType ct in availableContentTypes)
 116:                         {
 117:                             if (contentTypeGroup == null || ct.Group.ToLowerInvariant() == contentTypeGroup)
 118:                             {
 119:                                 contentTypes.Add(ct);
 120:                             }
 121:                         }
 122:                     }
 123:  
 124:                     if (Params["includefielddefinitions"].UserTypedIn)
 125:                     {
 126:                         // If we're including field definitions then we want to show them first as they'll need to appear first when using within a Feature
 127:                         foreach (SPContentType ct in contentTypes)
 128:                         {
 129:                             SPContentType parentCT = ct.Parent;
 130:                             foreach (SPField field in ct.Fields)
 131:                             {
 132:                                 // If the parent content type contains the current field and the user wants to exclude parent fields then continue to the next field.
 133:                                 if (parentCT != null && excludeParentFields && parentCT.Fields.ContainsField(field.InternalName))
 134:                                     continue;
 135:  
 136:                                 if (!ctFields.ContainsKey(field.Id))
 137:                                     ctFields.Add(field.Id, field);
 138:                             }
 139:                         }
 140:                         xmlWriter.WriteString("\r\n\r\n");
 141:                         foreach (SPField field in ctFields.Values)
 142:                         {
 143:                             string schema = field.SchemaXml;
 144:                             if (field.InternalName.Contains(ENCODED_SPACE) && removeEncodedSpaces)
 145:                             {
 146:                                 schema = schema.Replace(string.Format("Name=\"{0}\"", field.InternalName),
 147:                                                         string.Format("Name=\"{0}\"", field.InternalName.Replace(ENCODED_SPACE, string.Empty)));
 148:                             }
 149:  
 150:                             xmlWriter.WriteRaw(schema);
 151:                             xmlWriter.WriteString("\r\n");
 152:                         }
 153:                     }
 154:  
 155:                     xmlWriter.WriteString("\r\n");
 156:  
 157:                     foreach (SPContentType ct in contentTypes)
 158:                     {
 159:                         WriteContentTypeXml(ct, excludeParentFields, removeEncodedSpaces, xmlWriter);
 160:                     }
 161:  
 162:                     if (Params["includelistbindings"].UserTypedIn)
 163:                         GetListBindings(web, contentTypes, xmlWriter);
 164:  
 165:                     xmlWriter.WriteEndElement(); // Elements
 166:                 }
 167:                 finally
 168:                 {
 169:                     xmlWriter.Flush();
 170:                     xmlWriter.Close();
 171:                 }
 172:             }
 173:         }
 174:  
 175:         File.WriteAllText(Params["outputfile"].Value, sb.ToString());
 176:         return 1;
 177:     }
 178:  
 179:  
 180:     #endregion
 181:  
 182:  
 183:     /// <summary>
 184:     /// Writes the content type XML.
 185:     /// </summary>
 186:     /// <param name="ct">The ct.</param>
 187:     /// <param name="excludeParentFields">if set to <c>true</c> [exclude parent fields].</param>
 188:     /// <param name="removeEncodedSpaces">if set to <c>true</c> [remove encoded spaces].</param>
 189:     /// <param name="xmlWriter">The XML writer.</param>
 190:     private static void WriteContentTypeXml(SPContentType ct, bool excludeParentFields, bool removeEncodedSpaces, XmlTextWriter xmlWriter)
 191:     {
 192:         SPContentType parentCT = ct.Parent;
 193:         XmlDocument xmlDoc = new XmlDocument();
 194:         xmlDoc.LoadXml(ct.SchemaXml);
 195:         XmlElement fieldRefsElement = xmlDoc.CreateElement("FieldRefs");
 196:         xmlDoc.DocumentElement.AppendChild(fieldRefsElement);
 197:         foreach (XmlElement field in xmlDoc.SelectNodes("//Fields/Field"))
 198:         {
 199:             // If the parent content type contains the current field and the user wants to exclude parent fields then continue to the next field.
 200:             if (parentCT != null && excludeParentFields && parentCT.Fields.ContainsField(field.GetAttribute("Name")))
 201:                 continue;
 202:  
 203:             XmlElement fieldRefElement = xmlDoc.CreateElement("FieldRef");
 204:             fieldRefElement.SetAttribute("ID", field.GetAttribute("ID"));
 205:  
 206:             string name = field.GetAttribute("Name");
 207:             if (name.Contains(ENCODED_SPACE) && removeEncodedSpaces)
 208:                 name = name.Replace(ENCODED_SPACE, string.Empty);
 209:  
 210:             fieldRefElement.SetAttribute("Name", name);
 211:             fieldRefsElement.AppendChild(fieldRefElement);
 212:         }
 213:         xmlDoc.DocumentElement.RemoveChild(xmlDoc.SelectSingleNode("//Fields"));
 214:  
 215:         xmlWriter.WriteString("\r\n");
 216:         xmlWriter.WriteRaw(xmlDoc.OuterXml);
 217:     }
 218:  
 219:     /// <summary>
 220:     /// Gets the list bindings.
 221:     /// </summary>
 222:     /// <param name="web">The web.</param>
 223:     /// <param name="contentTypes">The content types.</param>
 224:     /// <param name="xmlWriter">The XML writer.</param>
 225:     private static void GetListBindings(SPWeb web, List<SPContentType> contentTypes, XmlTextWriter xmlWriter)
 226:     {
 227:         foreach (SPList list in web.Lists)
 228:         {
 229:             foreach (SPContentType listCT in list.ContentTypes)
 230:             {
 231:                 foreach (SPContentType ct in contentTypes)
 232:                 {
 233:                     //if ((listCT.Scope != ct.Scope && listCT.Parent.Id == ct.Id) || listCT.Id == ct.Id)
 234:                     if (listCT.Name == ct.Name && listCT.Group == ct.Group)
 235:                     {
 236:                         xmlWriter.WriteStartElement("ContentTypeBinding");
 237:                         xmlWriter.WriteAttributeString("ContentTypeId", ct.Id.ToString());
 238:                         xmlWriter.WriteAttributeString("ListUrl", list.RootFolder.ServerRelativeUrl);
 239:                         xmlWriter.WriteEndElement(); // ContentTypeBinding
 240:                     }
 241:                 }
 242:             }
 243:         }
 244:  
 245:         foreach (SPWeb subWeb in web.Webs)
 246:         {
 247:             try
 248:             {
 249:                 GetListBindings(subWeb, contentTypes, xmlWriter);
 250:             }
 251:             finally
 252:             {
 253:                 subWeb.Dispose();
 254:             }
 255:         }
 256:     }
 257: }
 258:  
The syntax of the command can be seen below:
C:\>stsadm -help gl-exportcontenttypes

stsadm -o gl-exportcontenttypes


Exports Content Types to an XML file.

Parameters:
        -url <url containing the content types>
        -outputfile <file to output results to>
        [-name <name of an individual content type to export>]
        [-group <content type group name to filter results by>]
        [-listname <name of a list to export content types from>]
        [-includelistbindings]
        [-includefielddefinitions]
        [-excludeparentfields]
        [-removeencodedspaces (removes '_x0020_' in field names)]
Here's an example of how to dump the XMl for all the content types in a particular group:
stsadm -o gl-exportcontenttypes -url "http://intranet" -outputfile c:\contentTypes.xml -group "Custom Content Types" -includelistbindings -includefielddefinitions
8Nov/079

Export, Import, and Update List Fields

One thing that's been hanging over my head for a while is what to do about the Site Directory. The first problem was to get the business users to decide on where it should live (the master site directory that is) and what columns (or meta data) should be part of the directory (either new columns or changes to existing columns). Once I finally got that information I had to solve my second problem which was to find a way to script all the changes. I've got commands for moving lists and list items but I had no ability to add new fields or update existing fields.

After looking at the SPField object some I discovered that the SchemaXml property has a getter and a setter and that I could add new fields via the SPFieldCollection.AddFieldAsXml() method which takes in the same XML that you can get via the SchemaXml property. With this information I concluded that I could very easily create commands to export, import, and update a list field.

The commands I created are: gl-exportlistfield, gl-importlistfield, and gl-updatelistfield. Now I could set up a field in my test environment, export the field out to an XML file and then import or update that field during the upgrade (depending on whether it's a new field or changes to an existing field). Fortunately the code to do all of this was extremely simple and it only took me a few minutes to create and test each one. The import and export are the simplest and least likely to throw errors - the update could cause you issues if you are attempting to make invalid changes (like changing the type from a Text type to a Choice type - I don't do any validation of your XML and rely completely on Microsoft's internal validation). The commands I created are detailed below.

1. gl-exportlistfield

Because I already had helper methods to get a field the code for this became basically just one line of text: File.WriteAllText(Params["outputfile"].Value, field.SchemaXml); The rest of the code is in a utility method which I've previously shown in other posts so won't show again here. The syntax of the command can be seen below:

C:\>stsadm -help gl-exportlistfield

stsadm -o gl-exportlistfield

Exports a list field (column) to a file.

Parameters:
        -url <list view URL>
        -fielddisplayname <field display name> / -fieldinternalname <field internal name>
        -outputfile <file to output field schema to>

Here's an example of how export the DivisionMulti field to a file:

stsadm –o gl-exportlistfield -url "http://intranet/SiteDirectory/SitesList/AllItems.aspx" -fieldinternalname DivisionMulti -outputfile "c:\divisionmulti.xml"

Running the above command will produce results similar to the following :

<Field Name="DivisionMulti" DisplayName="Division" Type="MultiChoice" ColName="ntext3" ID="{A96D82DA-601E-435B-9E8A-C086A853387B}" StaticName="DivisionMulti" SourceID="{93C04BF8-4E28-4194-A584-D6A97FCC87AE}">
 <CHOICES>
  <CHOICE>Information Technology</CHOICE>
  <CHOICE>Research &amp; Development</CHOICE>
  <CHOICE>Sales</CHOICE>
  <CHOICE>Finance</CHOICE>
 </CHOICES>
</Field> 

2. gl-importlistfield

Once we have our list field exported we can then import the field into another list either as is or with whatever manual modifications you may have made (just be careful that you know what you are doing - it's always better to make the modifications using the browser and then export those changes than it is to try and hack the XML directly).

By default when you run the import command the code will attempt to locate the "ID" attribute and replace it with a new GUID value - otherwise you may get errors stating that the field already exists (even if you've changed the "Name" attribute). If you don't want the code to do this then you can pass in the "-retainobjectidentity" parameter. Note that I'm not doing anything with the "SourceID" attribute as I couldn't detect any issues with keeping the value unchanged.

Like the export command there's really not much to the code - I get the SPList object, load up the XML, replace the ID attribute if needed, and then call AddFieldAsXml() and then call the ReorderField method (detailed at the end of this post in the 11/9/2007 update):

   1: public override int Run(string command, StringDictionary keyValues, out string output)
   2: {
   3:  output = string.Empty;
   4:  
   5:  InitParameters(keyValues);
   6:  
   7:  string url = Params["url"].Value;
   8:  string xml = File.ReadAllText(Params["inputfile"].Value);
   9:  SPAddFieldOptions fieldOptions = SPAddFieldOptions.Default;
  10:  if (Params["addfieldoptions"].UserTypedIn)
  11:   fieldOptions = (SPAddFieldOptions) Enum.Parse(typeof (SPAddFieldOptions), Params["addfieldoptions"].Value, true);
  12:  
  13:  XmlDocument xmlDoc = new XmlDocument();
  14:  xmlDoc.LoadXml(xml);
  15:  Guid id = new Guid(xmlDoc.DocumentElement.GetAttribute("ID"));
  16:  if (!Params["retainobjectidentity"].UserTypedIn)
  17:  {
  18:   id = Guid.NewGuid();
  19:   xmlDoc.DocumentElement.SetAttribute("ID", id.ToString());
  20:   xml = xmlDoc.OuterXml;
  21:  }
  22:  
  23:  SPList list = Utilities.GetListFromViewUrl(url);
  24:  list.Fields.AddFieldAsXml(xml, Params["addtodefaultview"].UserTypedIn, fieldOptions);
  25:  
  26:  SPField field = list.Fields[id];
  27:  if (Params["sortindex"].UserTypedIn)
  28:  {
  29:   int sortIndex = int.Parse(Params["sortindex"].Value);
  30:   ReorderField(list, field, sortIndex);
  31:  }
  32:  
  33:  return 1;
  34: }

The syntax of the command can be seen below:

stsadm -o gl-importlistfield

Imports a field (column) into a list.

Parameters:
        -url <list view URL>
        -inputfile <input file containing field schema information>
        [-addfieldoptions <default | addtodefaultcontenttype | addtonocontenttype | addtoallcontenttypes | addfieldinternalnamehint | addfieldtodefaultview | addfieldcheckdisplayname>
        [-addtodefaultview]
        [-retainobjectidentity]
        [-sortindex <field order index>]

Here's an example of how to import the field exported above into a new list (note that it assumes that the DivisionMulti field does not exist in the target list and that there's no field with a display name of "Division")and setting the sort order index to zero thus making it the first item in the list:

stsadm –o gl-importlistfield -url "http://teamsites/sitedirectory/siteslist/allitems.aspx" -inputfile "c:\divisionmulti.xml" -addfieldoptions addfieldcheckdisplayname -addtodefaultview -sortindex 0

3. gl-updatelistfield

When I first set out to create this command I originally thought it was going to be a real pain in the @$$ but then I discovered that the SchemaXml property had a setter and life got a whole lot easier. What I thought would end up being hundreds of lines of code to deal with all the possible changes ended up being one core line plus a few others just to get the data. I had to a bit of additional complexity though to deal with the ability to determine the field to edit via the name parameters and then to deal with the fact that I wanted to be able to edit the sort order without having to change anything else (thus making the inputfile optional):

   1: public override int Run(string command, StringDictionary keyValues, out string output)
   2: {
   3:  output = string.Empty;
   4:  
   5:  InitParameters(keyValues);
   6:  
   7:  string url = Params["url"].Value;
   8:  string fieldTitle = Params["fielddisplayname"].Value;
   9:  string fieldName = Params["fieldinternalname"].Value;
  10:  bool useTitle = Params["fielddisplayname"].UserTypedIn;
  11:  bool useName = Params["fieldinternalname"].UserTypedIn;
  12:  bool inputFileProvided = Params["inputfile"].UserTypedIn;
  13:  
  14:  
  15:  if (!inputFileProvided && !Params["sortindex"].UserTypedIn)
  16:  {
  17:   throw new SPSyntaxException("You must either specify an input file with changes or a sort index.");
  18:  }
  19:  if (!inputFileProvided && !useTitle && !useName)
  20:  {
  21:   throw new SPSyntaxException(
  22:    "You must specify either an input file with changes or the field name to update.");
  23:  }
  24:  
  25:  SPList list;
  26:  SPField field;
  27:  XmlDocument xmlDoc = new XmlDocument();
  28:  string xml = null;
  29:  
  30:  if (inputFileProvided)
  31:  {
  32:   xml = File.ReadAllText(Params["inputfile"].Value);
  33:   xmlDoc.LoadXml(xml);
  34:  }
  35:  
  36:  if (!inputFileProvided || useTitle || useName)
  37:  {
  38:   field = Utilities.GetField(url, fieldName, fieldTitle, useName, useTitle);
  39:   list = field.ParentList;
  40:  }
  41:  else
  42:  {
  43:   list = Utilities.GetListFromViewUrl(url);
  44:   string internalName = xmlDoc.DocumentElement.GetAttribute("Name");
  45:   field = list.Fields.GetFieldByInternalName(internalName);
  46:  }
  47:  
  48:  if (inputFileProvided)
  49:  {
  50:   field.SchemaXml = xml;
  51:   field.Update();
  52:  }
  53:  
  54:  if (Params["sortindex"].UserTypedIn)
  55:  {
  56:   int sortIndex = int.Parse(Params["sortindex"].Value);
  57:   ImportListField.ReorderField(list, field, sortIndex);
  58:  }
  59:  
  60:  return 1;
  61: }

The syntax of the command can be seen below:

C:\>stsadm -help gl-updatelistfield

stsadm -o gl-updatelistfield

Updates a field (column) using the provided input XML.  Use exportlistfield to get the existing schema and then modify the results (note that the 'Name' attribute of the Field node must not change unless the fieldinternalname or fielddisplayname is passed in as this attribute is what is used to determine which field to update).

Parameters:
        -url <list view URL>
        [-inputfile <input file containing the field schema information>]
        [-fielddisplayname <field display name> / -fieldinternalname <field internal name>]
        [-sortindex <field order index>]

Here's an example of how to update the field exported above with changes made to the resultant XML file (such as adding new choice elements) and setting the sort order index to zero thus making it the first item in the list:

stsadm –o gl-updatelistfield -url "http://intranet/sitedirectory/siteslist/allitems.aspx" -inputfile "c:\divisionmulti.xml" -sortindex 0

I thought about using the "ID" attribute within the XML to locate the field to update but in the end I decided that someone may want to use this to change the ID for whatever reason and they're less likely to want to change the internal name.

 

Update 11/9/2007: I've modified the gl-updatelistfield and gl-importlistfield commands so that they now support the passing in of a sortindex parameter which effectively does what it says - it changes the field order. I've updated the content above to reflect the changes. I wish that this were one of those easy things to implement but it turns out that it was a real pain - fortunately I found a post by Michael Ekegren which discusses how to do this using the ProcessBatchData method of the SPWeb object. I've included the code that makes this work below:

   1: /// <summary>
   2: /// Reorders the field.
   3: /// </summary>
   4: /// <param name="list">The list.</param>
   5: /// <param name="field">The field.</param>
   6: /// <param name="sortIndex">The sort index.</param>
   7: internal static void ReorderField(SPList list, SPField field, int sortIndex)
   8: {
   9:  if (field.Reorderable)
  10:  {
  11:   List<SPField> fields = new List<SPField>();
  12:   int count = 0;
  13:   bool added = false;
  14:   // First add the reorderable fields
  15:   for (int i = 0; i < list.Fields.Count; i++)
  16:   {
  17:    if (list.Fields[i].Reorderable)
  18:    {
  19:     if (count == sortIndex)
  20:     {
  21:      added = true;
  22:      fields.Add(field);
  23:      count++;
  24:     }
  25:  
  26:     if (list.Fields[i].Id == field.Id)
  27:      continue;
  28:  
  29:     fields.Add(list.Fields[i]);
  30:     count++;
  31:    }
  32:   }
  33:   if (!added)
  34:    fields.Add(field);
  35:  
  36:   // Now add the non-reorderable fields
  37:   for (int i = 0; i < list.Fields.Count; i++)
  38:   {
  39:    if (!list.Fields[i].Reorderable)
  40:    {
  41:     fields.Add(list.Fields[i]);
  42:    }
  43:   }
  44:  
  45:   StringBuilder sb = new StringBuilder();
  46:  
  47:   XmlTextWriter xmlWriter = new XmlTextWriter(new StringWriter(sb));
  48:   xmlWriter.Formatting = Formatting.Indented;
  49:  
  50:   xmlWriter.WriteStartElement("Fields");
  51:  
  52:   for (int i = 0; i < fields.Count; i++)
  53:   {
  54:    xmlWriter.WriteStartElement("Field");
  55:    xmlWriter.WriteAttributeString("Name", fields[i].InternalName);
  56:    xmlWriter.WriteEndElement();
  57:   }
  58:  
  59:   xmlWriter.WriteEndElement();
  60:   xmlWriter.Flush();
  61:  
  62:   using (SPWeb web = list.ParentWeb)
  63:   {
  64:    ReorderFields(web, list, sb.ToString());
  65:   }
  66:  }
  67: }
  68:  
  69: /// <summary>
  70: /// This function reorders the fields in the specified list programmatically as specified by the xmlFieldsOrdered parameter
  71: /// </summary>
  72: /// <param name="web">The SPWeb object containing the list</param>
  73: /// <param name="list">The SPList object to update</param>
  74: /// <param name="xmlFieldsOrdered">A string in XML-format specifying the field order by the location within a xml-tree</param>
  75: private static void ReorderFields(SPWeb web, SPList list, string xmlFieldsOrdered)
  76: {
  77:  try
  78:  {
  79:   string fpRPCMethod = @"<?xml version=""1.0"" encoding=""UTF-8""?>
  80:    <Method ID=""0,REORDERFIELDS"">
  81:    <SetList Scope=""Request"">{0}</SetList>
  82:    <SetVar Name=""Cmd"">REORDERFIELDS</SetVar>
  83:    <SetVar Name=""ReorderedFields"">{1}</SetVar>
  84:    <SetVar Name=""owshiddenversion"">{2}</SetVar>
  85:    </Method>";
  86:  
  87:   // relookup list version in order to be able to update it
  88:   list = web.Lists[list.ID];
  89:  
  90:   int currentVersion = list.Version;
  91:  
  92:   string version = currentVersion.ToString();
  93:   string RpcCall = string.Format(fpRPCMethod, list.ID, SPHttpUtility.HtmlEncode(xmlFieldsOrdered), version);
  94:  
  95:   web.AllowUnsafeUpdates = true;
  96:  
  97:   web.ProcessBatchData(RpcCall);
  98:  }
  99:  catch (System.Net.WebException err)
 100:  {
 101:   Console.WriteLine("WARNING:" + err.Message);
 102:  }
 103: }
3Oct/071

Apply Upgrade Area Url Mappings

As I mentioned in my previous post (Replace Field Values) I'm currently trying to solve the issue of broken links throughout the various sites that I've upgraded and moved around. The gl-replacefieldvalues command was the first of three that I've created. This second command, called gl-applyupgradeareaurlmappings, is very similar to the gl-replacefieldvalues command but it is a bit specialized.

I've created this command to address the list items in the special Upgrade Area Url Mappings list (http://[portal]/Lists/98d3057cd9024c27b2007643c1/AllItems.aspx). Instead of taking in a search string and replace string like the other command does I use this list as the source for the search and replace string. The intent is to prevent the user from having to deal with the spsredirect.aspx page when clicking links within a web application.

It's important to note though that I've limited this command to be scoped at the web application level and below and the upgrade list must exist in the web application being targeted. I went back and forth on whether I wanted to allow a farm scope but in the end decided against it. I also considered allowing you to target one application but reference the list from another but the reality of that was that it would be much too difficult to do a reliable replace due to the fact that the URLs in the list are server relative so I simply wouldn't be able to know which to make absolute and which web app to use for the absolute path.

I haven't been able to do a significant amount of testing with this but it seems to work fairly well - I was concerned about the order in which the replacements occur which could result in funny things happening but in the test runs that I did the replacements seemed to work just fine (if anyone finds differently please let me know and I'll see if I can work it out).

Because of this and due to the general nature of making batch content replacements I strongly suggest using the "-test" parameter and verifying all the changes that will be attempted before you execute for real - if you don't and things get messed up then you'll have a nice new command to create: "revertpreviouscheckins" sounds like a good name :)

The large bulk of this code is just a series of methods with different loops in them to handle the various scoping capabilities. The core code itself is simply looking at all fields of type string and doing a regular expression replace using the information from the upgrade list. Like the gl-replacefieldvalues command I've added the ability to dump all changes to a log file as well as to run the command in a "test" mode where it will show you what it would change but not actually make the change (as mentioned above) - Again, I'd strongly recommend you use this first to verify all the changes that will be made. The core code is shown below:

   1: /// <summary>
   2: /// Replaces the values.
   3: /// </summary>
   4: /// <param name="list">The list.</param>
   5: /// <param name="settings">The settings.</param>
   6: private static void ReplaceValues(SPList list, Settings settings)
   7: {
   8:  if (list.Title.ToLowerInvariant() == "upgrade area url mapping" || list.DefaultViewUrl.ToLowerInvariant().IndexOf(UPGRADE_AREA_URL_LIST) >= 0)
   9:   return; // We don't want to process this list as it's the source of our data.
  10:  
  11:  Log(settings, "Processing List: " + list.DefaultViewUrl);
  12:  
  13:  if (upgradeUrls == null)
  14:  {
  15:   BuildReplacementUrlsList(settings);
  16:  }
  17:  
  18:  // We've got the replacement list built - now we need to check every field of every item.
  19:  foreach (SPListItem item in list.Items)
  20:  {
  21:   if (item.File != null && !Utilities.IsCheckedOutByCurrentUser(item))
  22:   {
  23:    continue;
  24:   }
  25:   bool wasCheckedOut = true;
  26:   bool modified = false;
  27:  
  28:   foreach (SPField field in list.Fields)
  29:   {
  30:    if (item[field.Id] == null || field.ReadOnlyField)
  31:     continue;
  32:  
  33:    Type fieldType = item[field.Id].GetType();
  34:  
  35:    if (fieldType != typeof(string))
  36:     continue; // We're only going to work with strings.
  37:  
  38:    string fieldName = field.Title.ToLowerInvariant();
  39:    if (settings.UseInternalFieldName)
  40:     fieldName = field.InternalName.ToLowerInvariant();
  41:  
  42:    if (settings.FieldName == null || settings.FieldName.ToLowerInvariant() == fieldName)
  43:    {
  44:      //  if (item[field.Id].ToString().IndexOf("C4/") >= 0)
  45:      //      System.Diagnostics.Debugger.Break();
  46:  
  47:     // We've found a field that we need to consider.  Check all possibilities to see if a match is found.
  48:     foreach (ReplacmentSettings rs in upgradeUrls)
  49:     {
  50:      bool isMatch = rs.Regex.IsMatch((string)item[field.Id]);
  51:  
  52:        // if (item[field.Id].ToString().IndexOf("C4/") >= 0 && rs.Regex.ToString().IndexOf("C4/") >= 0)
  53:        //     System.Diagnostics.Debugger.Break();
  54:  
  55:      if (!isMatch)
  56:       continue;
  57:  
  58:      string result = rs.Regex.Replace((string)item[field.Id], rs.ReplaceString);
  59:  
  60:      Log(settings,
  61:       string.Format("Match found: List={0}, Field={1}, Replacement={2} => {3}", item.Url,
  62:            field.Title, item[field.Id], result));
  63:  
  64:      if (!settings.Test)
  65:      {
  66:       if (item.File != null && item.File.CheckOutStatus == SPFile.SPCheckOutStatus.None)
  67:       {
  68:        item.File.CheckOut();
  69:        wasCheckedOut = false;
  70:       }
  71:       item[field.Id] = result;
  72:       modified = true;
  73:      }
  74:     }
  75:    }
  76:   }
  77:   if (modified && !settings.Test)
  78:    item.Update();
  79:  
  80:   if (modified && item.File != null)
  81:    item.File.CheckIn("Checking in changes to list item due to automated upgrade area url mappings being applied.");
  82:   else if (!wasCheckedOut && item.File != null)
  83:    item.File.UndoCheckOut();
  84:  
  85:   if (modified && settings.Publish && !wasCheckedOut)
  86:    PublishItems.PublishListItem(item, list,
  87:            new PublishItems.Settings(settings.Quiet, settings.Test,
  88:                    settings.LogFile),
  89:            "\"stsadm.exe -o replacefieldvalues\"");
  90:  }
  91:  Log(settings, "Finished Processing List: " + list.DefaultViewUrl + "\r\n");
  92: }
  93:  
  94: /// <summary>
  95: /// Builds the replacement urls list.
  96: /// </summary>
  97: /// <param name="settings">The settings.</param>
  98: private static void BuildReplacementUrlsList(Settings settings)
  99: {
 100:  // We need to build a local copy of the replacement urls so that we don't have to keep referring back
 101:  // to the list for the information.
 102:  upgradeUrls = new List<ReplacmentSettings>();
 103:  foreach (SPListItem item in settings.UpgradeList.Items)
 104:  {
 105:   if ((bool)item["ShouldRedirect"])
 106:   {
 107:    ReplacmentSettings rs = new ReplacmentSettings();
 108:    rs.ReplaceString = (string)item["V3ServerRelativeUrl"];
 109:    string str = (string) item["V2ServerRelativeUrl"];
 110:  
 111:    rs.Regex = new Regex("(?i:" + str + "|" + SPEncode.UrlEncodeAsUrl(str) + ")");
 112:  
 113:    upgradeUrls.Add(rs);
 114:   }
 115:  }
 116: }
The syntax of the command I created can be seen below.

C:\>stsadm -help gl-applyupgradeareaurlmappings

stsadm -o gl-applyupgradeareaurlmappings

Replaces all occurrences of the V2 server relative URLs with the corresponding V3 server relative URLs as specified in the Upgrade Area Url Mappings list: "http://[portal]/Lists/98d3057cd9024c27b2007643c1/AllItems.aspx"

Parameters:
        -url <url to search>
        -scope <WebApplication | Site | Web | List>
        [-field <field name>]
        [-useinternalfieldname (if not present then the display name will be used)]
        [-quiet]
        [-test]
        [-logfile <log file>]
        [-publish]

Here’s an example of how to replace all occurrences of the V2ServerRelativeUrls with the corresponding V3ServerRelativeUrl:

stsadm -o gl-applyupgradeareaurlmappings -url "http://intranet/" -scope webapplication -logfile "c:\replace.log" -publish

If you wish to filter by field name you can specify either the display name or the internal name. Note that if you specify the internal name you must also add the "useinternalfieldname" parameter - if you don't use the internal field name then be aware that you may be updating more than the field you intended as the display name is not always unique (but in most cases it is).

If you don't want your changes published then don't specify the publish parameter. However, specifying the publish parameter will not publish items that were previously checked out. If an item can be published and it requires approval then specifying the publish parameter will also cause the item to be approved.

2Oct/0718

Replace Field Values

Because I'm doing so much moving around of the post-upgraded sites I'm running into lots of issues with broken links. In an attempt to solve this problem I've begun working on 3 commands. This first command, gl-replacefieldvalues, will take in a search string and a replacement string and modify every list item where a match is made. The searching can be scoped to the entire farm, a single web application, a single site collection (and all sub-sites), a single web site (and all sub-sites) or a single list and can be further restricted to only update a specific field.

The command uses Regex.Replace() so you can get pretty complex with your replacements (I'm no regular expression expert so I've kept my uses fairly simplistic). The other two commands I'll document later (one will apply all the URL changes identified in the Upgrade Area Url Mappings list and the other will attempt to handle content within web parts - I'm still creating these and should be finished some time this week).

The large bulk of this code is just a series of methods with different loops in them to handle the various scoping capabilities. The core code itself is simply looking at all fields of type string and doing a regular expression replace using the provided details. I've also added the ability to dump all changes to a log file as well as to run the command in a "test" mode where it will show you what it would change but not actually make the change - I'd strongly recommend you use this first to verify all the changes that will be made. The core code is shown below:

   1: private static void ReplaceValues(SPList list, Settings settings)
   2: {
   3:  Log(settings, "Processing List: " + list.DefaultViewUrl);
   4:  
   5:  Regex regex = new Regex(settings.SearchString);
   6:  
   7:  foreach (SPListItem item in list.Items)
   8:  {
   9:   if (item.File != null && !Utilities.IsCheckedOutByCurrentUser(item))
  10:   {
  11:    continue;
  12:   }
  13:   bool wasCheckedOut = true;
  14:   bool modified = false;
  15:  
  16:   foreach (SPField field in list.Fields)
  17:   {
  18:    if (item[field.Id] == null || field.ReadOnlyField)
  19:     continue;
  20:  
  21:    if (list.Title == "98d3057cd9024c27b2007643c1" && field.Title == "V2ServerRelativeUrl")
  22:     continue; // We don't want to change this url because then external links will break.
  23:  
  24:    Type fieldType = item[field.Id].GetType();
  25:  
  26:    if (fieldType != typeof(string))
  27:     continue; // We're only going to work with strings.
  28:  
  29:    string fieldName = field.Title.ToLowerInvariant();
  30:    if (settings.UseInternalFieldName)
  31:     fieldName = field.InternalName.ToLowerInvariant();
  32:  
  33:    if (settings.FieldName == null || settings.FieldName.ToLowerInvariant() == fieldName)
  34:    {
  35:     bool isMatch = regex.IsMatch((string)item[field.Id]);
  36:     
  37:     if (!isMatch)
  38:      continue;
  39:     string result = regex.Replace((string) item[field.Id], settings.ReplaceString);
  40:  
  41:     Log(settings, string.Format("Match found: List={0}, Field={1}, Replacement={2} => {3}", item.Url, field.Title, item[field.Id], result));
  42:  
  43:     if (!settings.Test)
  44:     {
  45:      if (item.File != null && item.File.CheckOutStatus == SPFile.SPCheckOutStatus.None)
  46:      {
  47:       item.File.CheckOut();
  48:       wasCheckedOut = false;
  49:      }
  50:      item[field.Id] = result;
  51:      modified = true;
  52:     }
  53:    }
  54:   }
  55:   if (modified && !settings.Test)
  56:    item.Update();
  57:  
  58:   if (modified && item.File != null)
  59:    item.File.CheckIn("Checking in changes to list item due to automated search and replace (\"" + settings.SearchString + "\" replaced with \"" + settings.ReplaceString + "\").");
  60:   else if (!wasCheckedOut && item.File != null)
  61:    item.File.UndoCheckOut();
  62:  
  63:   if (modified && settings.Publish && item.File != null)
  64:    item.File.Publish("Publishing changes to list item due to automated search and replace (\"" + settings.SearchString + "\" replaced with \"" + settings.ReplaceString + "\").");
  65:  
  66:   if (modified && settings.Publish && item.ModerationInformation != null && item.File != null)
  67:    item.File.Approve("Approving changes to list item due to automated search and replace (\"" + settings.SearchString + "\" replaced with \"" + settings.ReplaceString + "\").");
  68:  
  69:  }
  70:  Log(settings, "Finished Processing List: " + list.DefaultViewUrl + "\r\n");
  71: }
The syntax of the command I created can be seen below.

C:\>stsadm -help gl-replacefieldvalues

stsadm -o gl-replacefieldvalues

Replaces all occurrences of the search string with the replacement string.  Supports use of regular expressions.  Use -test to verify your replacements before executing.

Parameters:
        [-url <url to search>]
        {-inputfile <input file> |
         -searchstring <regular expression string to search for>
         -replacestring <replacement string>}
        -scope <Farm | WebApplication | Site | Web | List>
        [-field <field name>]
        [-useinternalfieldname (if not present then the display name will be used)]
        [-inputfiledelimiter <delimiter character to use in the input file (default is "|")>]
        [-inputfileisxml (input is XML in the following format: <Replacements><Replacement><SearchString>string</SearchString><ReplaceString>string</ReplaceString></Replacement><Replacements>)
        [-quiet]
        [-test]
        [-logfile <log file>]
        [-publish]

Here’s an example of how to replace all occurrences of "/Topics/Divisions/HumanResources/" with "/hr/" in all lists within a web application using a case insensitive match:

stsadm -o gl-replacefieldvalues -url "http://intranet/" -scope webapplication -searchstring "(?i:/Topics/Divisions/HumanResources/)" -replacestring "/hr/" -logfile "c:\replace.log" -publish

If you wish to filter by field name you can specify either the display name or the internal name. Note that if you specify the internal name you must also add the "useinternalfieldname" parameter - if you don't use the internal field name then be aware that you may be updating more than the field you intended as the display name is not always unique (but in most cases it is).

If you don't want your changes published then don't specify the publish parameter. However, specifying the publish parameter will not publish items that were previously checked out. If an item can be published and it requires approval then specifying the publish parameter will also cause the item to be approved. Use of regular expressions can be extremely powerful and dangerous especially when updating at a large scope so again, I strongly encourage the use of the -test parameter to verify the changes to be made before you actually apply your changes.

Update 12/13/2007: I've updated this command to support the passing in of an input file containing search and replace strings. The input file can be either a flat file where each replacement is on a separate line with a delimiter separating the search and replace strings or an XML file. The syntax details have been updated above. Thanks to Glenn Hickman for helping with this.

14Sep/073

Delete List Field

One of the things I've been working on recently was trying to get the site directory into a useable state post upgrade. The main issue I needed to address was that the upgrade resulted in duplicate columns for Division and Region. The duplication was because the default Site Directory template creates columns with a display name of Division (and Region) but the internal name is DivisionMulti (and RegionMulti). So when the upgrade moved the source list over it saw the names as being differing so instead of replace or merging the columns it just added new ones (this time with the internal name matching the display name).

So, I needed a simple command to allow me to delete the DivisionMulti and RegionMulti columns. The problem that I encountered was that the indexer for the Fields collection of the list (SPFieldCollection.Item[string]) wants the display name - this threw me off at first and I can see where it could potentially mess someone up really bad if they're not paying close attention to how they are using it. By passing in only the display name you will be returned back the first item that it comes across with no indication whatsoever that there may actually be another item in the list with the same name.

So my challenge here was to make the command as "safe" as I could. I accomplished this by allowing the user to pass in either the display name or the internal name (with the internal name being the safest approach). If the display name was passed in then I do a check to make sure that there's only one field with that display name - if I find more than one then I return back the matches along with the internal name so that the command can be attempted again. I thought about only allowing the internal name but figured that 90% of the time the display name will work just fine and it's easier to discover.

The other thing I needed to do was to provide some details as to why a field may not be able to be deleted. To do this I had to look into the CanBeDeleted property of the SPField object. From this I was able to see that if the field is derived from a base type or if it's sealed or if it's marked as not allowing deletion via the AllowDeletion property then it cannot be deleted. The AllowDeletion property is an easy enough one to get around so I provided a "-force" switch to allow the user to override this setting and delete the field regardless. I chose not to do anything (not allow the deletion) if it's derived from a base type or if it's sealed only because of numerous ramifications that result from attempting to delete these fields.

There's only two core parts to this code - the first involves a utility method that I created to retrieve the SPField object and the other is a small amount of code to determine whether or not the field can be deleted or not:

   1: internal static SPField GetField(string listViewUrl, string fieldName, string fieldTitle, bool useFieldName, bool useFieldTitle)
   2: {
   3: SPList list = Utilities.GetListFromViewUrl(listViewUrl);
   4:  
   5: if (list == null)
   6: {
   7:     throw new Exception("List not found.");
   8: }
   9:  
  10: SPField field = null;
  11: try
  12: {
  13:     // It's possible to have more than one field in the fields collection with the same display name.  The Fields collection
  14:     // will merely return back the first item it finds with a name matching the one specified so it's really a rather useless
  15:     // way of retrieving a field as it's extremely misleading and could lead to someone inadvertantly messing things up.
  16:     // I provide the ability to use the display name for convienence but don't rely on it for anything.
  17:     if (useFieldTitle)
  18:         field = list.Fields[fieldTitle];
  19: }
  20: catch (ArgumentException)
  21: {
  22: }
  23:  
  24: if (field != null || useFieldName)
  25: {
  26:     int count = 0;
  27:     string foundFields = string.Empty;
  28:     // If the user specified the display name we need to make sure that only one field exists matching that display name.
  29:     // If they specified the internal name then we need to loop until we find a match.
  30:     foreach (SPField temp in list.Fields)
  31:     {
  32:         if (useFieldName && temp.InternalName.ToLower() == fieldName.ToLower())
  33:         {
  34:             field = temp;
  35:             break;
  36:         }
  37:         else if (useFieldTitle && temp.Title == fieldTitle)
  38:         {
  39:             count++;
  40:             foundFields += "\t" + temp.Title + " = " + temp.InternalName + "\r\n";
  41:         }
  42:     }
  43:     if (useFieldTitle && count > 1)
  44:     {
  45:         throw new Exception("More than one field was found matching the display name specified:\r\n\r\n\tDisplay Name = Internal Name\r\n\t----------------------------\r\n" +
  46:                             foundFields +
  47:                             "\r\nUse \"-fieldinternalname\" to delete based on the internal name of the field.");
  48:     }
  49: }
  50:  
  51: if (field == null)
  52:     throw new Exception("Field not found.");
  53: return field;
  54: }
  55:  
  56: public override int Run(string command, StringDictionary keyValues, out string output)
  57: {
  58:     output = string.Empty;
  59:  
  60:     InitParameters(keyValues);
  61:     SPBinaryParameterValidator.Validate("fielddisplayname", Params["fielddisplayname"].Value, "fieldinternalname", Params["fieldinternalname"].Value);
  62:  
  63:     string url = Params["url"].Value;
  64:     string fieldTitle = Params["fielddisplayname"].Value;
  65:     string fieldName = Params["fieldinternalname"].Value;
  66:     bool useTitle = Params["fielddisplayname"].UserTypedIn;
  67:     bool useName = Params["fieldinternalname"].UserTypedIn;
  68:     bool force = Params["force"].UserTypedIn;
  69:  
  70:     SPField field = Utilities.GetField(url, fieldName, fieldTitle, useName, useTitle);
  71:  
  72:     if (!field.CanBeDeleted)
  73:     {
  74:         if (field.FromBaseType)
  75:         {
  76:             throw new Exception(
  77:                 "The field is derived from a base type and cannot be deleted.  You must delete the field from the base type.");
  78:         }
  79:         else if (field.Sealed)
  80:         {
  81:             throw new Exception("This field is sealed and cannot be deleted.");
  82:         }
  83:         else if (field.AllowDeletion.HasValue && !field.AllowDeletion.Value && !force)
  84:         {
  85:             throw new Exception(
  86:                 "Field is marked as not allowing deletion - specify \"-force\" to ignore this setting and attempt deletion regardless.");
  87:         }
  88:         else if (field.AllowDeletion.HasValue && !field.AllowDeletion.Value && force)
  89:         {
  90:             field.AllowDeletion = true;
  91:             field.Update();
  92:         }
  93:         else
  94:         {
  95:             throw new Exception("Field cannot be deleted.");
  96:         }
  97:     }
  98:  
  99:     field.Delete();
 100:  
 101:     return 1;
 102: }

The syntax of the command I created, gl-deletelistfield, can be seen below:

C:\>stsadm -help gl-deletelistfield

stsadm -o gl-deletelistfield

Deletes a field (column) from a list.

Parameters:
        -url <list view URL>
        [-fielddisplayname <field display name>] / [-fieldinternalname <field internal name>]
        [-force (used to force deletion if AllowDeletion property is false)]

Here’s an example of how to delete the DivisionMulti column from the site directory:

stsadm –o gl-deletelistfield -url "http://intranet/sitedirectory/lists/sites/allitems.aspx" -fieldinternalname DivisionMulti