How Unreal Macro Generated

Environment
UnrealEngine branch: ue5-early-access
Visual Studio 2019 version: 16.10.4
Windows 10 build: 19043.1110

Expanding UCLASS()

1
2
3
4
/UnrealEngine/Engine/Source/Runtime/Engine/Classes/GameFramework/PlayerController.h

UCLASS(config=Game, BlueprintType, Blueprintable, meta=(ShortTooltip="A Player Controller is an actor responsible for controlling a Pawn used by the player."))
class ENGINE_API APlayerController : public AController

The macro UCLASS() may be the most famous one of the unreal macros. First of all, let us find out how it can be expanded. Our goal is expanding UCLASS() of the class APlayerController, which can be found at line #222 of PlayerController.h.

1
2
3
4
5
6
7
/UnrealEngine/Engine/Source/Runtime/CoreUObject/Public/UObject/ObjectMacros.h

#if UE_BUILD_DOCS || defined(__INTELLISENSE__ )
#define UCLASS(...)
#else
#define UCLASS(...) BODY_MACRO_COMBINE(CURRENT_FILE_ID,_,__LINE__,_PROLOG)
#endif

In UnrealEngine, most of definitions for core macros are placed in ObjectMacros.h file. We can see the definition of UCLASS here, and it would be the second definition in usual case. Then, what is the macro BODY_MACRO_COMBINE ?

1
2
3
4
5
/UnrealEngine/Engine/Source/Runtime/CoreUObject/Public/UObject/ObjectMacros.h

// This pair of macros is used to help implement GENERATED_BODY() and GENERATED_USTRUCT_BODY()
#define BODY_MACRO_COMBINE_INNER(A,B,C,D) A##B##C##D
#define BODY_MACRO_COMBINE(A,B,C,D) BODY_MACRO_COMBINE_INNER(A,B,C,D)

The macro is defined as BODY_MACRO_COMBINE_INNER, which concatenates parameters as one string. Thus, the macro UCLASS would result the text like below:

1
2
3
4
UCLASS(...) ->
BODY_MACRO_COMBINE(CURRENT_FILE_ID,_,222,_PROLOG) ->
BODY_MACRO_COMBINE_INNER(CURRENT_FILE_ID,_,222,_PROLOG) ->
CURRENT_FILE_ID_222_PROLOG

It can be tested with simple code. Check it out at the screenshot below. The __LINE__ is one of pre-defined macro, so it is turned to 222, where the UCLASS is written.

Here is test code and its result. Check the name of integer variable.

Actually, the macro CURRENT_FILE_ID can be found at header files generated by Unreal Header Tool. You can find the definition at generated header files, for instance, PlayerController.generated.h. The generated header files are created when you attempt to build your project.

1
2
3
4
/UnrealEngine/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Engine/PlayerController.generated.h

#undef CURRENT_FILE_ID
#define CURRENT_FILE_ID Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h

Thus, the macro CURRENT_FILE_ID would be replaced before BODY_MACRO_COMBINE is expanded. We can rewrite the macro evaluation process.

1
2
3
4
UCLASS(...) ->
BODY_MACRO_COMBINE(Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h,_,222,_PROLOG) ->
BODY_MACRO_COMBINE_INNER(Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h,_,222,_PROLOG) ->
Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_222_PROLOG

And, the Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_222_PROLOG is also defined at the generated header file for PlayerController.h.

1
2
3
4
/UnrealEngine/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Engine/PlayerController.generated.h

#define Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_222_PROLOG \
	Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_225_EVENT_PARMS

The Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_225_EVENT_PARMS is a macro containing definitions for essential structures.

1
2
3
4
5
6
7
8
9
10
11
/UnrealEngine/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Engine/PlayerController.generated.h

#define Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_225_EVENT_PARMS \
	struct PlayerController_eventClientAddTextureStreamingLoc_Parms \
	{ \
		FVector InLoc; \
		float Duration; \
		bool bOverrideLocation; \
	}; \
	struct PlayerController_eventClientCapBandwidth_Parms \
    ...

Let us rewrite the evaluation process. As a result, the macro UCLASS is replaced by definitions for some essential structures.

1
2
3
4
5
6
UCLASS(...) ->
BODY_MACRO_COMBINE(Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h,_,222,_PROLOG) ->
BODY_MACRO_COMBINE_INNER(Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h,_,222,_PROLOG) ->
Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_222_PROLOG ->
Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_225_EVENT_PARMS ->
struct PlayerController_eventClientAddTextureStreamingLoc_Parms...(omitted)

Okay, we have just peeled off one layer to the truth. But, is that all ? We should take care of something more…Most of time, the macro UCLASS is not solely used. Various keywords and specifiers come with this. (ex: config=Game, BlueprintType, meta=(ShortTooltip=..., …) So, how they are handled ? Even, how the generated header file is created ?

Generated Header File

1
2
3
4
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/CodeGenerator.cpp

FPreloadHeaderFileInfo& FileInfo = PreloadedFiles[Index];
bool bHasChanged = ConstThis->WriteHeader(FileInfo, GeneratedHeaderText, AdditionalHeaders, ReferenceGatherers, TempSaveTasks[Index]);

The UHT writes header files containing auto-generated codes at the code above. The PreloadedFiles has absolute paths of generated header file, for instance, D:/Git/UnrealEngine/Engine/.../Inc/Engine/PlayerController.generated.h.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/CodeGenerator.cpp

TArray<FPreloadHeaderFileInfo> PreloadedFiles;
PreloadedFiles.SetNum(Exported.Num());

ParallelFor(Exported.Num(), [&Exported, &PreloadedFiles, Package=Package, ConstPackageManifest](int32 Index)
{
	FUnrealSourceFile* SourceFile = Exported[Index];

	FString ModuleRelativeFilename = SourceFile->GetFilename();
	ConvertToBuildIncludePath(Package, ModuleRelativeFilename);

	FString StrippedName = FPaths::GetBaseFilename(MoveTemp(ModuleRelativeFilename));
	FString HeaderPath = (ConstPackageManifest->GeneratedIncludeDirectory / StrippedName) + TEXT(".generated.h");

	PreloadedFiles[Index].Load(MoveTemp(HeaderPath));
});

An example of NoExportTypes.generated.h.

You can track what UHT writes on the generated header file via the variable GeneratedHeaderText. Because its contents will replace old generated header file, whenever there is any difference between old contents and new contents.

1
2
3
4
5
6
7
8
9
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/CodeGenerator.cpp

GeneratedHeaderText.Logf(
	TEXT("#ifdef %s")																	LINE_TERMINATOR
	TEXT("#error \"%s.generated.h already included, missing '#pragma once' in %s.h\"")	LINE_TERMINATOR
	TEXT("#endif")																		LINE_TERMINATOR
	TEXT("#define %s")																	LINE_TERMINATOR
	LINE_TERMINATOR,
	*FileDefineName, *StrippedFilename, *StrippedFilename, *FileDefineName);

For example, there is some definition for preventing duplicated include. The format above turns to like below.

1
2
3
4
5
6
/UnrealEngine/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Engine/PlayerController.generated.h

#ifdef ENGINE_PlayerController_generated_h
#error "PlayerController.generated.h already included, missing '#pragma once' in PlayerController.h"
#endif
#define ENGINE_PlayerController_generated_h

There are more things worthy to check.

1
2
3
4
5
6
7
8
9
10
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/CodeGenerator.cpp

FString MacroName = SourceFile.GetGeneratedMacroName(ClassData, TEXT("_EVENT_PARMS"));
WriteMacro(OutGeneratedHeaderText, MacroName, UClassMacroContent);
PrologMacroCalls.Logf(TEXT("\t%s\r\n"), *MacroName);

...

GeneratedHeaderText.Log(TEXT("#undef CURRENT_FILE_ID\r\n"));
GeneratedHeaderText.Logf(TEXT("#define CURRENT_FILE_ID %s\r\n\r\n\r\n"), *SourceFile->GetFileId());

That is why the CURRENT_FILE_ID and ..._EVENT_PARMS macros are defined. Furthermore, other codes can be found at FNativeClassHeaderGenerator::FNativeClassHeaderGenerator(const UPackage*, const TSet<FUnrealSourceFile*>&, FClasses&, bool).

So, we have found the relationship of unreal macro and generated header file. But, there is one thing left, the metadata.

Metadata Parser

1
2
3
4
/UnrealEngine/Engine/Source/Runtime/Engine/Classes/GameFramework/PlayerController.h

UCLASS(config=Game, BlueprintType, Blueprintable, meta=(ShortTooltip="A Player Controller is an actor responsible for controlling a Pawn used by the player."))
class ENGINE_API APlayerController : public AController

Back to the start, there are metadata within the macro UCLASS such as config=Game, BlueprintType and meta=.... We are going to check out how they are handled by UnrealEngine.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
/UnrealEngine/Engine/Source/Runtime/CoreUObject/Public/UObject/ObjectMacros.h

// These are used for syntax highlighting and to allow autocomplete hints

namespace UC
{
	// valid keywords for the UCLASS macro
	enum 
	{
		/// This keyword is used to set the actor group that the class is show in, in the editor.
		classGroup,

		/// Declares that instances of this class should always have an outer of the specified class.  This is inherited by subclasses unless overridden.
		Within, /* =OuterClassName */

		/// Exposes this class as a type that can be used for variables in blueprints
		BlueprintType,

		/// Prevents this class from being used for variables in blueprints
		NotBlueprintType,

		/// Exposes this class as an acceptable base class for creating blueprints. The default is NotBlueprintable, unless inherited otherwise. This is inherited by subclasses.
		Blueprintable,

		/// Specifies that this class is *NOT* an acceptable base class for creating blueprints. The default is NotBlueprintable, unless inherited otherwise. This is inherited by subclasses.
		NotBlueprintable,
		...

You can find some enum definitions seems related to the metadata. But, they are for only supporting autocomplete hints such as Intellisense and VisualAssistX. There is another code handling metadata.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/UnrealEngine/Engine/Source/Programs/UnrealHeader/Private/CodeGenerator.cpp

const TArray<FString>& UObjectHeaders =
	(CurrentlyProcessing == PublicClassesHeaders) ? Module.PublicUObjectClassesHeaders :
	(CurrentlyProcessing == PublicHeaders       ) ? Module.PublicUObjectHeaders        :
							Module.PrivateUObjectHeaders;
...
ParallelFor(UObjectHeaders.Num(), [&](int32 Index)
{
	const FString& RawFilename = UObjectHeaders[Index];

#if !PLATFORM_EXCEPTIONS_DISABLED
	try
#endif
	{
		PerformSimplifiedClassParse(Package, *RawFilename, *HeaderFiles[Index], PerHeaderData[Index]);
	}
...

This code is for parsing metadata. Any header file in your project is passed through the parsing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
static void SetupUObjectModuleHeader(UHTModuleInfo ModuleInfo, FileItem HeaderFile, SourceFileMetadataCache MetadataCache)
{
	// Check to see if we know anything about this file.  If we have up-to-date cached information about whether it has
	// UObjects or not, we can skip doing a test here.
	if (MetadataCache.ContainsReflectionMarkup(HeaderFile))
	{
		lock(ModuleInfo)
		{
			bool bFoundHeaderLocation = false;
			foreach (DirectoryReference ModuleDirectory in ModuleInfo.ModuleDirectories)
			{
				if (HeaderFile.Location.IsUnderDirectory(DirectoryReference.Combine(ModuleDirectory, "Classes")))
				{
					ModuleInfo.PublicUObjectClassesHeaders.Add(HeaderFile);
					bFoundHeaderLocation = true;
				}
				else if (HeaderFile.Location.IsUnderDirectory(DirectoryReference.Combine(ModuleDirectory, "Public")))
				{
					ModuleInfo.PublicUObjectHeaders.Add(HeaderFile);
					bFoundHeaderLocation = true;
				}
			}
			if (!bFoundHeaderLocation)
			{
				ModuleInfo.PrivateUObjectHeaders.Add(HeaderFile);
			}
		}
	}
}

About all modules, header files in Classes folder are stored at PublicUObjectClassesHeaders and header files in Public folder are stored at PublicUObjectHeaders. Even you have located a header file in other folder, the Unreal Build Tool collects it into PrivateUObjectHeaders.

A screenshot on debugging UBT.

Back to the FBaseParser::ReadSpecifierSetInsideMacro(), let us test with the keyword BlueprintType. How does the BlueprintType keyword parsed ? The UHT parses your header file with tokens. Suppose the input as UCLASS(config=Game, BlueprintType, Blueprintable, meta=(...)).

1
2
3
4
5
6
7
8
9
10
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/HeaderParser.cpp

if (Token.Matches(TEXT("UCLASS"), ESearchCase::CaseSensitive))
{
	bHaveSeenUClass = true;
	bEncounteredNewStyleClass_UnmatchedBrackets = true;
	UClass* Class = CompileClassDeclaration(AllClasses);
	GStructToSourceLine.Add(Class, MakeTuple(GetCurrentSourceFile()->AsShared(), Token.StartLine));
	return true;
}

Due to this code, the left input would be (config=Game, BlueprintType, Blueprintable, meta=(...)). And, the following tokenizing is like below based on FBaseParser::ReadSpecifierSetInsideMacro().

1
2
3
4
5
6
7
8
9
10
(config=Game, BlueprintType, Blueprintable, meta=(...))
-> RequireSymbol(TEXT('('), ErrorMessageGetter);
config=Game, BlueprintType, Blueprintable, meta=(...))
-> GetToken(Specifier); SpecifiersFound.Emplace(Specifier.Identifier);
, BlueprintType, Blueprintable, meta=(...))
-> RequireSymbol(TEXT(','), ErrorMessageGetter);
BlueprintType, Blueprintable, meta=(...))
-> GetToken(Specifier); GetMetadataKeyword(Specifier.Identifier);
, Blueprintable, meta=(...))
...

A screenshot on debugging UHT.

Wrap-Up

1
2
3
4
5
6
7
8
C/CPP (pure) macro
C/CPP code with macro ---(preprocessor)--->
C/CPP code with evaluated code from macro ---(rest of job)---> ...

unreal macro
C/CPP code with unreal macro ---(UHT and UBT)--->
C/CPP code with generated code(+macro) from UHT and UBT ---(preprocessor)--->
C/CPP code with evaluated code from macro ---(rest of job)---> ...

There are so many hidden code for implementing unreal macros, and the macros have complicated relationship with other engine code. Even most part of final code from the macros cannot be evaluated before some preprocessing and compilation. In this perspective, unreal macro such as UCLASS is not a pure C/CPP macro, because unreal macro functions fully only when UHT and UBT must preprocess the macro.

There is no doubt. Any of C/CPP compiler cannot recognize the unreal macro such as UCLASS. Even the Epic Games did not modify the compilers, and did not have to do. They have simply setup some build pipeline satisfying their needs. The program managing their custom build pipeline is the Unreal Build Tool, UBT. Most of jobs for build are done by UBT and UHT. In official document for UHT, these background knowledge is introduced.

1
2
3
4
5
6
UnrealHeaderTool (UHT) is a custom parsing and code-generation tool that supports the UObject system. Code compilation happens in two phases:

1. UHT is invoked, which parses the C++ headers for Unreal-related class metadata and generates custom code to implement the various UObject-related features.
2. The normal C++ compiler is invoked to compile the results.

When compiling, it is possible for either tool to emit errors, so be sure to look carefully.

As they said, the compilation order is the opposite direction of the paragraphs; Expanding UCLASS, Generated Header File and Metadata Parser. The actions for Expanding UCLASS are done by C/CPP compilers(+preprocessors), and the actions for Generated Header File and Metadata Parser are done by UHT. Additionally, actions for Metadata Parser happens early than ones for Generated Header File.

1
2
main function for parsing metadata -> FHeaderParser::ParseHeaders()
main function for generating header file -> FHeaderParser::ExportNativeHeaders()

Let us make a conclusion.

  • The result from unreal macro is hard to evaluate before processing by UBT(+UHT).
  • Some features of UnrealEngine are implemented by auto-generated codes.
  • You should look into the build pipeline of UnrealEngine if need to modify unreal macro things.