Incremental T4 Text Templating

Dispatches From The Build Tooling Front Lines

Since starting at Microsoft earlier this year I’ve done a lot of maintenance work on my team’s build system. One of the things I’ve been working on has been incremental building: if you run a build in Visual Studio, and then run another build straight away, it should detect that nothing has changed and skip rebuilding. Incremental building is very important for day-to-day productivity, especially in large codebases, since rebuilding one project usually forces all of its downstream dependents to rebuild too. (The devs in my department typically have around 60 projects loaded into Visual Studio; the whole repo consists of around 1300 projects(!). So as you can imagine a cascading incremental build failure can take quite some time and be a major productivity drag.)

Anyway, here’s one of the incremental build issues I fixed. It’s a bit outside my usual wheelhouse but I wanted to write it down because I couldn’t find anything online when I was first investigating this issue.

Debugging the Up-To-Date Check

I noticed that my team’s project wasn’t building incrementally: whenever I tried to re-run a unit test I had to wait for the whole project to recompile. The first thing to do, when you notice yourself feeling annoyed by recompilation, is to enable Up-To-Date Check Logging in Visual Studio. (I recommend just leaving the logging on “minimal” mode at all times.) The “Up-To-Date Check” is the component of Visual Studio which is responsible for incremental builds. It works by looking at the timestamps of the project’s input and output files; if all of the output files are newer than all of the input files then the project doesn’t need to be rebuilt.

After turning on the logging and rebuilding, I saw this message in the Output window:

FastUpToDate: Input Compile item 'CodeGen\Templates\AdapterTextTemplate.cs' is newer than earliest output 'bin\net472\ScopeCompiler.dll', not up-to-date.

That AdapterTextTemplate.cs file is generated using T4 Text Templating. We’d set T4 up to run as part of the build, so we didn’t need to check in the generated files and you didn’t have to manually run T4 before building. We did this by setting TransformOnBuild in the csproj file, and adding a Target to include any newly generated cs files:

<PropertyGroup>
    <!--
        In MSBuild parlance, a "property" is a scalar (string) value.
        Anything appearing inside a `PropertyGroup` is a property.
        The `TransformOnBuild` property tells T4 to run at build time.
    -->
    <TransformOnBuild>true</TransformOnBuild>
</PropertyGroup>

<ItemGroup>
    <!--
        MSBuild manages lists of "items"; an item roughly corresponds
        to a file. Inside an `ItemGroup` you can add items to a list
        using `Include`. So this line of code adds all of the `tt`
        files in the `CodeGen\Templates` folder to the list of
        `T4Transform` items.
    -->
    <T4Transform Include="CodeGen\Templates\*.tt" />

    <!-- Make the .tt files visible in Visual Studio -->
    <None Include="CodeGen\Templates\*.tt" />
</ItemGroup>

<!-- This `targets` file contains code to read the `T4Transform` item list and invoke T4. -->
<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v$(VisualStudioVersion)\TextTemplating\Microsoft.TextTemplating.targets" />

<!--
    A "target" is a coarse-grained unit of work —
    a stage in MSBuild's pluggable pipeline.

    This target runs in between the `ExecuteTransformations`
    target (from `Microsoft.TextTemplating.targets`) and the
    standard `Compile` target. It adds any output files that
    were generated by T4 (the `GeneratedFiles` item list)
    to the list of `Compile` items.
-->
<Target Name="IncludeT4GeneratedFiles"
        AfterTargets="ExecuteTransformations"
        BeforeTargets="Compile">

    <ItemGroup>
        <!-- Don't include already included files -->
        <Compile Include="@(GeneratedFiles)" Exclude="@(Compile)" />
    </ItemGroup>
</Target>

I realised that the incremental build failure was due to an interaction between build-time text templating and MSBuild’s default includes. The Up-To-Date Check scans the source tree for any cs files; if any of them have changed (or been created) since the last time a build ran, then the project needs rebuilding. Then, during the build (before compilation), T4 runs and generates some cs files; by default these cs files are placed alongside the tt file — ie, in the source tree. The next time you run a build, Visual Studio sees that the generated cs files have been touched, meaning the project needs to be rebuilt, meaning that T4 needs to run, which touches the generated cs files…

The Fix

For the fix I made a few changes:

  1. I excluded the generated cs files from the default list of files to track in the Up-To-Date Check.
  2. Instead, I included them dynamically, after T4 has run. (Visual Studio doesn’t look at dynamically added items when determining whether a project needs rebuilding.)
  3. I configured MSBuild to run T4 only if the input tt files have changed.

Excluding the Generated Files

One way to do this would be to tell T4 to put the generated files in the obj directory by default:

<!-- An `ItemDefinitionGroup` contains default metadata definitions for items. -->
<ItemDefinitionGroup>
    <T4Transform>
        <OutputFilePath>$(MSBuildProjectDirectory)$(IntermediateOutputPath)</OutputFilePath>
    </T4Transform>
    <T4Preprocess>
        <OutputFilePath>$(MSBuildProjectDirectory)$(IntermediateOutputPath)</OutputFilePath>
    </T4Preprocess>
</ItemDefinitionGroup>

(As far as I can tell the OutputFilePath metadata is not documented anywhere; I found it by decompiling the TransformTemplatesBase task in Microsoft.TextTemplating.Build.Tasks.dll.)

This probably would’ve been the cleaner way to do it, as it doesn’t pollute the source tree with generated files, but I didn’t want to break my teammates’ workflow by making it harder to find the generated code. Instead, I simply removed the generated files’ Compile items:

<ItemGroup>
    <!-- by convention, an underscore denotes private implementation details -->
    <_T4OutputFile Include="@(T4Preprocess -> '%(RelativeDir)%(Filename).cs')" />
    <_T4OutputFile Include="@(T4Transform -> '%(RelativeDir)%(Filename).cs')" />
    <Compile Remove="@(_T4OutputFile)" />
</ItemGroup>

T4 can generate any text file, not just cs files, so you’d need to tweak this code if your project happens to have T4 templates which generate other files.

Including the Generated Files Dynamically

Top-level properties and items are evaluated up-front, at the start of a build, but you can also manipulate properties and item lists dynamically by nesting them inside a Target. So all I had to do was plug in to the TransformDuringBuild target (implemented in Microsoft.TextTemplating.targets) and include the _T4OutputFiles after they’d been generated — much like the IncludeT4GeneratedFiles target I mentioned earlier.

I could’ve used AfterTargets="TransformDuringBuild" but instead I decided to override the TransformDuringBuild target. (To override a target, you just define a new target with the same name — last one wins. Overriding targets is usually not a good idea, but I had a good reason which will shortly become apparent.) The standard TransformDuringBuild target is defined like this:

<Target
    Name="TransformDuringBuild"
    Condition="'$(TransformOnBuild)' == 'true'"
    BeforeTargets="CoreCompile"
    DependsOnTargets="TransformAll">
</Target>

It’s a no-op target which simply causes another target (TransformAll) to be run when the TransformOnBuild property is set. So, for my override, I pasted that code and added an ItemGroup to the target’s body:

<Target
    Name="TransformDuringBuild"
    Condition="'$(TransformOnBuild)' == 'true'"
    BeforeTargets="CoreCompile"
    DependsOnTargets="TransformAll">

    <ItemGroup>
        <Compile Include="@(_T4OutputFile)">
            <AutoGen>True</AutoGen>
            <DesignTime>True</DesignTime>
        </Compile>
        <!--
            Add `FileWrites` items in order to make the generated
            files get deleted by Visual Studio's "clean" operation
        -->
        <FileWrites Include="@(_T4OutputFile)" />
    </ItemGroup>
</Target>

Running T4 Incrementally

MSBuild supports incremental building at the target level (so, finer-grained than a whole project). You can tell MSBuild to skip a target if its input files haven’t changed, using the Inputs and Outputs attributes. If all of the files listed in the Outputs are newer than the Inputs, then MSBuild will skip the target to save time. (Any items and properties defined in the target will still be included — it caches them from the last time the target was run.) This is why I decided to override the TransformDuringBuild target above — I wanted to give it Inputs and Outputs.

My final TransformDuringBuild target looked like this:

<Target
    Name="TransformDuringBuild"
    Condition="'$(TransformOnBuild)' == 'true'"
    BeforeTargets="CoreCompile"
    DependsOnTargets="TransformAll"
    Inputs="@(T4Preprocess);@(T4Transform)"
    Outputs="@(_T4OutputFile)">

    <ItemGroup>
        <Compile Include="@(_T4OutputFile)">
            <AutoGen>True</AutoGen>
            <DesignTime>True</DesignTime>
        </Compile>
        <FileWrites Include="@(_T4OutputFile)" />
    </ItemGroup>
</Target>

The Code

I put all of this code in a file called TextTemplating.CSharp.targets and imported it into the projects which used T4.

Here is the final file in handy pasteable form:

<Project>

    <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v$(VisualStudioVersion)\TextTemplating\Microsoft.TextTemplating.targets" />

    <ItemGroup>
        <Service Include="{508349b6-6b84-4df5-91f0-309beebad82d}" />
        <AvailableItem Include="T4Preprocess" />
        <AvailableItem Include="T4Transform" />
    </ItemGroup>

    <ItemGroup>
        <_T4OutputFile Include="@(T4Preprocess -> '%(RelativeDir)%(Filename).cs')" />
        <_T4OutputFile Include="@(T4Transform -> '%(RelativeDir)%(Filename).cs')" />

        <!--
            Don't include the generated files before they've been generated
            as it causes incremental build failure. Instead, include them after
            running T4, in the (overridden) TransformDuringBuild target
        -->
        <Compile Remove="@(_T4OutputFile)" />

        <!-- make visible in visual studio -->
        <None Include="@(T4Preprocess);@(T4Transform)" />
        <None Include="@(_T4OutputFile)" />
    </ItemGroup>

    <!-- Override the target from MS.TextTemplating.targets in order to set the Inputs/Outputs -->
    <Target
        Name="TransformDuringBuild"
        Condition="'$(TransformOnBuild)' == 'true'"
        BeforeTargets="CoreCompile"
        DependsOnTargets="TransformAll"
        Inputs="@(T4Preprocess);@(T4Transform)"
        Outputs="@(_T4OutputFile)">

        <ItemGroup>
            <Compile Include="@(_T4OutputFile)">
                <AutoGen>True</AutoGen>
                <DesignTime>True</DesignTime>
            </Compile>
            <FileWrites Include="@(_T4OutputFile)" />
        </ItemGroup>
    </Target>

</Project>

That’s the general recipe when you need to generate something at build time: write a Target with Inputs and Outputs. Put an ItemGroup inside the target to inform the rest of the build pipeline about the newly generated bits.