My Home Network Setup Experience (2021)

Prologue

About 4 months ago, I moved to new house, which is rented for 2 years. The building had been built in 2018, so I expected a quite simple and modern facilities including home networks infra. But, on the day moving to new house, a previous tenant said to me that “Only one LAN port works while others not”. At that time, I took this as a misunderstanding of the previous tenant. Because it is common that general people cannot handle or solve an network issue easily. Well…as you can see I write this post, he was right.

The previous tenant has mostly used the internet via wireless network, a.k.a. Wi-fi. It seemed that he does not know computer things. Even he connected to other rooms with exposed LAN cables. (I did not take the picture, but it was similar to a picture below.)

Floor Plan

I made a floor plan for new house. The green markers mean LAN ports. The LAN port with blue check mark was the only one working properly. Other LAN ports were not. The orange marker means a terminal box. When I first opened the terminal box, it looked like a picture below.

Very weird. The red cable might be the inbound. But other cables are connected in disorder. In this situation, I cannot guess which cable is destinated to certain LAN port. So I followed steps below for examination.

  • ​Check whether the red cable is inbound. The result was yes.
  • Connect a inbound cable to each cable. Check where each cable is connected to.
  • Repeat the second step until all of unknown ports found.

After some moments, I could organize a mapping for LAN ports. Let us see the picture below.

My Goal

Now, preparation done. It was time to do my plan. My plan was…

  • Reuse my gears as possible. At that time, I had a wireless router and some switch hubs.
  • Enable 2 ports. The port #3 and port #4.
  • Activate wireless network at appropriate position.

For this purpose, I planned to put a switch hub into the terminal box. Then, the wireless router should be near port #3. Because the Wi-fi signal would get weak when the router is in terminal box. However, on trying this plan, I found a very critical problem.

First Try

There is no power socket in the terminal box. In other words, there is no way to place a switch hub in the terminal box. In general, switch hub consumes power even it is small amount. What a panic !

I searched for bypassing the issue. Fortunately, there is one way fits in my case. The PoE, Power over Ethernet. The PoE is usually used at certain devices such as CCTV, Network Router, and VoIP Phone. These devices can be installed restricted environments.

  • There may be no power socket or power source due to small space.
  • There may be only LAN cable due to intra structure.

Yes. That is a perfect feature for this situation.

  • I could use only LAN cables in the terminal box.
  • I did not want to lay the power socket as the house is rented.

So, I chose to find injector and splitter. They are needed to implement PoE infra. The injector injects signal and power into LAN cable. The splitter splits it into signal and power at destination. Therefore, the floor plan can be redrawn as below.

Though it looks like some mess, anyway it worked. First, injector provides power to splitter via Ethernet. Therefore, splitter can supply power to switch hub. Second, inbound signal gets distributed by switch hub in terminal box. Finally, home network is constructed by the router on port #3.

  • I reused my gears. The wireless router and switch hubs.
  • Now I can access internet via port #3 and port #4.
  • I activated wireless network at the middle of house, living room.

Great. I was satisfied with the result…for a while.

Second Try

It was totally fine that connecting multiple devices with the router. Of course, because the router is extremely close. But, the problem happened when I had added several devices on port #4 side. When using one device on port #4, the device could recognize the network well. In contrast, when using two devices on port #4, one of them could not recognize the network. It can be drawn as below.

By the way, the devices were too far from the router. There were two switch hub between the router and devices, and it could prone some network conflicts. I had decided to change my home network configuration, with keeping my goals mentioned before.

A solution for this problems is simple. Placing a router in the terminal box. Then, my home network would be like picture above. But, one thing left behind, a wireless network. I cannot expect a wireless network functions well if the wireless router is in terminal box because the terminal box must be closed.

So, I had to buy new router for putting it in the terminal box. Placing new router in the terminal box and leave old router in the same position would be okay. In this case, the old router must be used like switch hub, not a router. My home network looks like picture above. And I will be able to connect the devices on port #4.

Third Try

Hmm…everything works well. Nothing malfunctions. But Wi-fi SSID issue was annoying me. New router and old router had been activated on wireless network, and they got each one of SSID. Yeah, right. There are TWO SSID separately, even though in the same network. I wanted them to merge into one.

Searched again. Maybe the EasyMesh a solution for me. The EasyMesh is one of Wi-fi technology that enables to merge multiple access points. Even EasyMesh does not care about type of frequency of access point. In other words, all of access points with 2.4GHz or 5.0GHz frequency will be merged into single access point. That was what I looked for !

My routers were made by EFM networks, the company famous of ipTIME trademark. EFM networks provides an utility program controls EasyMesh configuration like picture above. (Almost every company seems that develops and implements the EasyMesh specification, so find out other companies too.)

Setting up EasyMesh was completely easy. The structure of my home network was fit for the conditions. Now I can access Wi-fi via only single SSID. Furthermore, Wi-fi range is larger than before by merging two SSID. What a convenient :)

Epilogue

With the series of effort, I could setup my home network completely without great expense. It was lucky that there was no need to call workers. Maybe I would tell a next tenant of these story and give advice. I do not want anyone to suffer from the problems. ;)

Oh, I almost forgot. You should check the router before buying it if you want to setup EasyMesh. EasyMesh requires two types of router; MeshController and MeshAgent. Check the item you gonna buy whether it is kind of MeshController or MeshAgent.

How to setup Perforce server with Ubuntu

Environment
Helix Core (P4D) version: P4D/LINUX26X86_64/2021.1/2156517
Ubuntu version: 20.04 LTS
Helix Visual Client (P4V) version: P4V/NTX64/2021.3/2170446
Windows 10 build: 19043.1165
References
Helix Core Server Administrator Guide https://www.perforce.com/manuals/p4sag/Content/P4SAG/chapter.install.html
.p4ignore for UnrealEngine https://github.com/mattmarcin/ue4-perforce/blob/master/.p4ignore
P4 Typemap for UnrealEngine https://docs.unrealengine.com/4.27/en-US/ProductionPipelines/SourceControl/Perforce/

Prepare Ubuntu Instance

As the Perforce is a kind of CVCS*, it is recommended to use a dedicated server. The dedicated server should run all day and night, and should have a fixed IP address. Thus, you would better choose a cloud server if not have a machine for the purposes. In this post, I chose DigitalOcean for a cloud server provider. The DigitalOcean provides instances with cheaper cost than others such as AWS. You know, you may not need a high quality instance for your small size project.
※ CVCS : Centralized Version Control System. Find more information at here.

Sign-up and Sign-in the DigitalOcean. Click the button Create and choose Droplets.

At the page Create Droplets, you would be asked to choose options for a instance. In this post, we gonna choose options like below:

  • Choose an image
    • Ubuntu 20.04 (LTS) x64
  • Choose a plan
    • SHARED CPU
      • Basic
    • CPU options
      • Regular Intel with SSD
      • $5/month (= $0.007/hour)
        • 1 core CPU, 1 GB RAM, 25 GB SSD, 1000 GB transfer
  • Add block storage
    • $5/month (= $0.007/hour)
      • 50 GB SSD
  • Choose a datacenter region
    • (Select a datacenter that is closest to your location)
    • (In my case, it is Singapore)
  • Select additional options
    • Monitoring
  • Authentication
    • Password
      • (Choose a password for entering the instance)

Any options I did not mention are left as default selection. Let us create our Droplet by clicking the button Create Droplet.

Now you can see the new Droplet. Connect the instance via SSH. You can do it with Powershell or WSL in Windows 10. The X.X.X.X must be replaced with the IP address of instance.

1
> ssh root@X.X.X.X

But, when you attempt to connect the instance via SSH, you are asked to enter a password.

1
root@X.X.X.X's password:

Enter the password that you typed at the Authentication text block. If the right password entered, you can see the logs like below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-73-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

System information as of Sat Sep 11 16:12:19 UTC 2021

System load: 0.0 Users logged in: 0
Usage of /: 9.3% of 24.06GB IPv4 address for eth0: X.X.X.X
Memory usage: 24% IPv4 address for eth0: X.X.X.X
Swap usage: 0% IPv4 address for eth1: X.X.X.X
Processes: 113

66 updates can be applied immediately.
1 of these updates is a standard security update.
To see these additional updates run: apt list --upgradable


*** System restart required ***
Last login: Sat Sep 11 11:35:09 2021 from X.X.X.X
root@ubuntu-test:~#

At the your instance page, you can check the volume setting. Click the Config Instructions.

If you selected the option Automatically Format & Mount at the section Add block storage, the volume is already attached even you did nothing. In other words, the process Mount the volume is already done. Let us check whether the volume is well mounted. The volume_X must be replaced with yours.

1
2
3
> cd /mnt/volume_X
> ls
lost+found

You successfully setup an Ubuntu instance. Good to go !

Install & Setup P4D

We need to setup public key for accessing Perforce packages. For this, you need to download the public key at https://package.perforce.com/perforce.pubkey. The download can be done by command below:

1
> curl https://package.perforce.com/perforce.pubkey > perforce.pubkey

This command let you save the public key as a file, whose name is perforce.pubkey.

1
2
3
4
5
6
> curl https://package.perforce.com/perforce.pubkey > perforce.pubkey
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1707 100 1707 0 0 2024 0 --:--:-- --:--:-- --:--:-- 2022
> ls
perforce.pubkey snap

You can check the contents of file with cat.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
> cat perforce.pubkey
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2.0.14 (GNU/Linux)

mQINBFIOq14BEADUC4gm+gjS/E/y2xXouALvMuK2xO/8nbcXJCUAD6Bi1xxmyDaR
LXHDJ5lIzZV8/Jctck2bBIWE1WE8Qfpfz/eAU5lJoQTovt0OkOnyyAyFBSk9yXtN
fscQGdTXkl9LVVfsaTHVT3WGZF+iMCIZOVjYGjqRh3ozZp3LWOQl3cwgOZKQCi9Q
y/YRn6XZIOiQQEfvLzrBL1oyD1BoOq8Y2CrwTfhyz93qIRu089mAr7lo2e6UM/KV
JRjk6rPFcKIE0aOP2UwgY/6LMeK65MAKib76EFbygXBprz9K5zwq70A7MGSPjRPw
A7kdzw53flZyNscI2c093jW/PkeDw4++01QFky/FFqJncjIHoOid42NvQXD/+E5e
JKhqYReS2eHpv5qgsSc2Febd5Ccd0B4+2ryY3MBXqaj759NH6uWAowHjAv90y4Cb
c2FugNBAJ6XQBQaXcWsfPnWpBFYL36LxBCcu+ddiycTWS9SWFT3h0FIgTbTQNNQr
Fgjg5vYAw+TWU8wf1I3sak+wbU25h7ErKN1oSJ/EbPwUFOc6zjaDUlnIgmCnQgEj
NdrlWGGfgCfYTHZTnGW6fGlpByDnYO0wn/okPJxRStnkbqb8QGKRa1uTObSM0/4J
aqsaReo2E45x2TAIY6rNuiLet/r1hZzpLs3dffvoddscb/LshW1eiNU36QARAQAB
tERQZXJmb3JjZSBTb2Z0d2FyZSAoUGFja2FnZSBTaWduaW5nKSA8c3VwcG9ydCtw
YWNrYWdpbmdAcGVyZm9yY2UuY29tPokCPgQTAQIAKAUCUg6rXgIbAwUJEswDAAYL
CQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQcSPLdg/xiGlcKxAAvkzIPVzhc2am
GaAAUse4mQ6InNqCLiGEbNxPWnd13myGesbESfyBxex5Gb2t47gVllA7P9hOcvsv
J6g56WPD1yb+5Wrdchdn6SkSEfg1MOAMTskFPPJJ/3ZgfHKn/kv3tOJPcQsidRFl
uqNMHSroHpOExYaTgB7IhYBjnYHLwUgH1ikCFgkRzdaDW4Qfx6IRB1vpSjzxCjzP
Cc78cf4VDmBdSfwsO6/ON19ZcxtLjHvQK5sz91qsEJdJZjyq6YCHYfP+Zx8/M55S
ixZ6/QsLRAsUYGuBjuWMpMgXjB06TXVbSg97bZt2tHBMZJ2OEMgn09eysyS9uwlY
HMtpu23jDTn6sKlRE6PYbZQirt2Ydq56wOQqJzrW1BbadX56DY8FLYBh8H9kcBaT
MGT1fiLfaEn1C8dG3D9aHdaPXZu+zGcMrPY+GMcObAAk3ICRXR20NknSByOxNEyz
nOqCsmr1nRdrpnf4+52G53xYKroWVRYeDdBukC8ik6weFjK4qy7C4ujOe1AmoBis
g6+R0huka1TYr9r94um+idvHniLnaZvnxPKEUMLnGesx9LYio4slqQj+nN6fIelv
onltXR49hpuAiOKtYUISmsk+rI+ep60DfDSQGbrV8HkW6KjuHjmE7EJKEEnIk5N/
JJfimlBk+rNgbcz0fpT6IDdS6PEoGwk=
=fj1X
-----END PGP PUBLIC KEY BLOCK-----

Let us register the public key.

1
2
3
4
5
6
> gpg --with-fingerprint perforce.pubkey
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: WARNING: no command supplied. Trying to guess what you mean ...
pub rsa4096 2013-08-16 [SC] [expires: 2023-08-14]
uid Perforce Software (Package Signing) <support+packaging@perforce.com>

Add the Perforce packaging key to your APT keyring.

1
2
> wget -qO - https://package.perforce.com/perforce.pubkey | sudo apt-key add -
OK

Execute the command for adding Perforce to your APT configuration.

1
2
3
4
5
6
7
8
9
10
11
> sudo add-apt-repository 'deb http://package.perforce.com/apt/ubuntu focal release'
Hit:1 https://repos.insights.digitalocean.com/apt/do-agent main InRelease
Hit:2 https://repos-droplet.digitalocean.com/apt/droplet-agent main InRelease
Get:3 http://mirrors.digitalocean.com/ubuntu focal InRelease [265 kB]
Hit:4 http://mirrors.digitalocean.com/ubuntu focal-updates InRelease
Hit:5 http://mirrors.digitalocean.com/ubuntu focal-backports InRelease
Get:6 http://package.perforce.com/apt/ubuntu focal InRelease [3650 B]
Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease
Get:8 http://package.perforce.com/apt/ubuntu focal/release amd64 Packages [8168 B]
Fetched 277 kB in 1s (298 kB/s)
Reading package lists... Done

Run update.

1
2
3
4
5
6
7
8
9
10
> apt-get update
Hit:1 https://repos.insights.digitalocean.com/apt/do-agent main InRelease
Get:2 http://mirrors.digitalocean.com/ubuntu focal InRelease [265 kB]
Hit:3 https://repos-droplet.digitalocean.com/apt/droplet-agent main InRelease
Hit:4 http://mirrors.digitalocean.com/ubuntu focal-updates InRelease
Hit:5 http://mirrors.digitalocean.com/ubuntu focal-backports InRelease
Hit:6 http://package.perforce.com/apt/ubuntu focal InRelease
Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease
Fetched 265 kB in 1s (321 kB/s)
Reading package lists... Done

Install the package.

1
2
3
4
5
6
7
8
9
> sudo apt-get install helix-p4d
Reading package lists... Done
Building dependency tree
Reading state information... Done
...
Started 0 services.
No services configured.
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.6) ...

Now you have one last step, launching the Perforce service ! Execute the batch file for it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
> sudo /opt/perforce/sbin/configure-helix-p4d.sh

Summary of arguments passed:

Service-name [(not specified)]
P4PORT [(not specified)]
P4ROOT [(not specified)]
Super-user [(not specified)]
Super-user passwd [(not specified)]
Unicode mode [(not specified)]
Case-sensitive [(not specified)]

For a list of other options, type Ctrl-C to exit, and then run:
$ sudo /opt/perforce/sbin/configure-helix-p4d.sh --help


You have entered interactive configuration for p4d. This script
will ask a series of questions, and use your answers to configure p4d
for first time use. Options passed in from the command line or
automatically discovered in the environment are presented as defaults.
You may press enter to accept them, or enter an alternative.

Please provide the following details about your desired Perforce environment:


Perforce Service name [master]:

You will be asked to enter some configurations such as name of service, directory, case sensitiviy, and so on. Setup them appropriately.

1
2
3
Perforce Service name [master]: Test
Service Test not found. Creating...
Perforce Server root (P4ROOT) [/opt/perforce/servers/Test]:

You should select the proper directory. It would be better to select the attached volume if the size of your project would be more than 25GB.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Perforce Service name [master]: Test
Service Test not found. Creating...
Perforce Server root (P4ROOT) [/opt/perforce/servers/Test]:
Create directory? (y/n) [y]: y
Perforce Server unicode-mode (y/n) [n]: y
Perforce Server case-sensitive (y/n) [y]:
Perforce Server address (P4PORT) [ssl:1666]:
Perforce super-user login [super]:
Perforce super-user password:
Re-enter password.
Perforce super-user password:

Configuring p4d service 'Test' with the information you specified...

Perforce db files in '/opt/perforce/servers/Test/root' will be created if missing...
...
::
:: - For help with creating Perforce Helix user accounts, populating
:: the depot with files, and making other customizations for your
:: site, see the Helix Versioning Engine Administrator Guide:
::
:: https://www.perforce.com/perforce/doc.current/manuals/p4sag/index.html
::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Now you can access the Perforce service via P4V at a client. Before that, take care of typemap. The typemap is an abbreviation of Type Mapping. You can define how Perforce handles certain type of files by it. If your project uses UnrealEngine, for the types related to UnrealEngine, Epic Games recommends to setup like below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
TypeMap:
binary+w //depot/....exe
binary+w //depot/....dll
binary+w //depot/....lib
binary+w //depot/....app
binary+w //depot/....dylib
binary+w //depot/....stub
binary+w //depot/....ipa
binary //depot/....bmp
text //depot/....ini
text //depot/....config
text //depot/....cpp
text //depot/....h
text //depot/....c
text //depot/....cs
text //depot/....m
text //depot/....mm
text //depot/....py
binary+l //depot/....uasset
binary+l //depot/....umap
binary+l //depot/....upk
binary+l //depot/....udk
binary+l //depot/....ubulk

You can edit the typemap of instance by executing command p4 typemap. The command would open typemap file with vi editor.

Add the Epic Games’s mappings to your mapping file*. Now all preparation of server side completed.
※ FYI, the // string does not mean “It is a kind of comment.” in P4 typemap system. You should copy the all of text.

Install & Setup P4V

Download the P4V installer at https://www.perforce.com/downloads/helix-visual-client-p4v and install with default options. When the installation compeleted, execute the P4V. You will see the display.

Enter ssl:X.X.X.X:1666 at the section Server. The X.X.X.X must be replaced with the IP address of instance. Enter super at the section User. The user super is an administrator account we have set. Now click the button OK. Check Trust this fingerprint and click the button Connect if you encounter the dialog like below:

Enter the password you set while launching the Perforce service at instance.

You can see the display when successfully entered.

The admin tool can be accessed at Tools/Administration.

In the tool, you can add or delete user directly.

Let us prepare some Depot and Stream. Click the Depots. Right-click any depot and select New Depot....

Type the name of new Depot.

Select stream at the section Deopt type and click OK.

Close the admin tool and return to the P4V*. Now you can find the new Depot at Depot view.
※ Actually, the admin tool was the program P4Admin, which is different with P4V. Just, Perforce supports to launch the P4Admin from P4V.

Restart the P4V for applying changes from P4Admin. After restart, find the File/New/Stream... and click it.

Let us make a Stream, name of mainline. The Stream will be placed in the new Deopt. Click the button OK.

Click New Workspace... at workspace view.

Name the new workspace and click the button Browse in line of Stream. You can find the Stream mainline at the dialog. Select it.

Finally, we have prepared a workspace in totally empty new Perforce service ! But, you should config p4ignore before getting into the work. Open any terminal and execute the command below:

1
> p4 set P4IGNORE=.p4ignore

This command will let your Perforce refer the file whose name is .p4ignore. It is kind of configuration lets you can use Perforce like git, which provides .gitignore. To apply this changes, restart P4V. And, create .p4ignore at your workspace directory. Let us test whether .p4ignore works well. Fill the contents of .p4ignore like below:

1
*.sln

Select Mark for Add... for .p4ignore and submit it.

Next, create a empty file whose name is Test.sln. Try to add this !

You try to check out the file, but the dialog would be popped-up. Great, your p4ignore works well.

If your project uses UnrealEngine, you should search for good one. I recommend you to use the p4ignore mentioned at references. Okay, then…all of preparation done. You are good to go :) !

How Unreal Macro Generated

Environment
UnrealEngine branch: ue5-early-access
Visual Studio 2019 version: 16.10.4
Windows 10 build: 19043.1110

Expanding UCLASS()

1
2
3
4
/UnrealEngine/Engine/Source/Runtime/Engine/Classes/GameFramework/PlayerController.h

UCLASS(config=Game, BlueprintType, Blueprintable, meta=(ShortTooltip="A Player Controller is an actor responsible for controlling a Pawn used by the player."))
class ENGINE_API APlayerController : public AController

The macro UCLASS() may be the most famous one of the unreal macros. First of all, let us find out how it can be expanded. Our goal is expanding UCLASS() of the class APlayerController, which can be found at line #222 of PlayerController.h.

1
2
3
4
5
6
7
/UnrealEngine/Engine/Source/Runtime/CoreUObject/Public/UObject/ObjectMacros.h

#if UE_BUILD_DOCS || defined(__INTELLISENSE__ )
#define UCLASS(...)
#else
#define UCLASS(...) BODY_MACRO_COMBINE(CURRENT_FILE_ID,_,__LINE__,_PROLOG)
#endif

In UnrealEngine, most of definitions for core macros are placed in ObjectMacros.h file. We can see the definition of UCLASS here, and it would be the second definition in usual case. Then, what is the macro BODY_MACRO_COMBINE ?

1
2
3
4
5
/UnrealEngine/Engine/Source/Runtime/CoreUObject/Public/UObject/ObjectMacros.h

// This pair of macros is used to help implement GENERATED_BODY() and GENERATED_USTRUCT_BODY()
#define BODY_MACRO_COMBINE_INNER(A,B,C,D) A##B##C##D
#define BODY_MACRO_COMBINE(A,B,C,D) BODY_MACRO_COMBINE_INNER(A,B,C,D)

The macro is defined as BODY_MACRO_COMBINE_INNER, which concatenates parameters as one string. Thus, the macro UCLASS would result the text like below:

1
2
3
4
UCLASS(...) ->
BODY_MACRO_COMBINE(CURRENT_FILE_ID,_,222,_PROLOG) ->
BODY_MACRO_COMBINE_INNER(CURRENT_FILE_ID,_,222,_PROLOG) ->
CURRENT_FILE_ID_222_PROLOG

It can be tested with simple code. Check it out at the screenshot below. The __LINE__ is one of pre-defined macro, so it is turned to 222, where the UCLASS is written.

Here is test code and its result. Check the name of integer variable.

Actually, the macro CURRENT_FILE_ID can be found at header files generated by Unreal Header Tool. You can find the definition at generated header files, for instance, PlayerController.generated.h. The generated header files are created when you attempt to build your project.

1
2
3
4
/UnrealEngine/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Engine/PlayerController.generated.h

#undef CURRENT_FILE_ID
#define CURRENT_FILE_ID Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h

Thus, the macro CURRENT_FILE_ID would be replaced before BODY_MACRO_COMBINE is expanded. We can rewrite the macro evaluation process.

1
2
3
4
UCLASS(...) ->
BODY_MACRO_COMBINE(Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h,_,222,_PROLOG) ->
BODY_MACRO_COMBINE_INNER(Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h,_,222,_PROLOG) ->
Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_222_PROLOG

And, the Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_222_PROLOG is also defined at the generated header file for PlayerController.h.

1
2
3
4
/UnrealEngine/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Engine/PlayerController.generated.h

#define Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_222_PROLOG \
Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_225_EVENT_PARMS

The Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_225_EVENT_PARMS is a macro containing definitions for essential structures.

1
2
3
4
5
6
7
8
9
10
11
/UnrealEngine/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Engine/PlayerController.generated.h

#define Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_225_EVENT_PARMS \
struct PlayerController_eventClientAddTextureStreamingLoc_Parms \
{ \
FVector InLoc; \
float Duration; \
bool bOverrideLocation; \
}; \
struct PlayerController_eventClientCapBandwidth_Parms \
...

Let us rewrite the evaluation process. As a result, the macro UCLASS is replaced by definitions for some essential structures.

1
2
3
4
5
6
UCLASS(...) ->
BODY_MACRO_COMBINE(Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h,_,222,_PROLOG) ->
BODY_MACRO_COMBINE_INNER(Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h,_,222,_PROLOG) ->
Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_222_PROLOG ->
Engine_Source_Runtime_Engine_Classes_GameFramework_PlayerController_h_225_EVENT_PARMS ->
struct PlayerController_eventClientAddTextureStreamingLoc_Parms...(omitted)

Okay, we have just peeled off one layer to the truth. But, is that all ? We should take care of something more…Most of time, the macro UCLASS is not solely used. Various keywords and specifiers come with this. (ex: config=Game, BlueprintType, meta=(ShortTooltip=..., …) So, how they are handled ? Even, how the generated header file is created ?

Generated Header File

1
2
3
4
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/CodeGenerator.cpp

FPreloadHeaderFileInfo& FileInfo = PreloadedFiles[Index];
bool bHasChanged = ConstThis->WriteHeader(FileInfo, GeneratedHeaderText, AdditionalHeaders, ReferenceGatherers, TempSaveTasks[Index]);

The UHT writes header files containing auto-generated codes at the code above. The PreloadedFiles has absolute paths of generated header file, for instance, D:/Git/UnrealEngine/Engine/.../Inc/Engine/PlayerController.generated.h.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/CodeGenerator.cpp

TArray<FPreloadHeaderFileInfo> PreloadedFiles;
PreloadedFiles.SetNum(Exported.Num());

ParallelFor(Exported.Num(), [&Exported, &PreloadedFiles, Package=Package, ConstPackageManifest](int32 Index)
{
FUnrealSourceFile* SourceFile = Exported[Index];

FString ModuleRelativeFilename = SourceFile->GetFilename();
ConvertToBuildIncludePath(Package, ModuleRelativeFilename);

FString StrippedName = FPaths::GetBaseFilename(MoveTemp(ModuleRelativeFilename));
FString HeaderPath = (ConstPackageManifest->GeneratedIncludeDirectory / StrippedName) + TEXT(".generated.h");

PreloadedFiles[Index].Load(MoveTemp(HeaderPath));
});

An example of NoExportTypes.generated.h.

You can track what UHT writes on the generated header file via the variable GeneratedHeaderText. Because its contents will replace old generated header file, whenever there is any difference between old contents and new contents.

1
2
3
4
5
6
7
8
9
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/CodeGenerator.cpp

GeneratedHeaderText.Logf(
TEXT("#ifdef %s") LINE_TERMINATOR
TEXT("#error \"%s.generated.h already included, missing '#pragma once' in %s.h\"") LINE_TERMINATOR
TEXT("#endif") LINE_TERMINATOR
TEXT("#define %s") LINE_TERMINATOR
LINE_TERMINATOR,
*FileDefineName, *StrippedFilename, *StrippedFilename, *FileDefineName);

For example, there is some definition for preventing duplicated include. The format above turns to like below.

1
2
3
4
5
6
/UnrealEngine/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Engine/PlayerController.generated.h

#ifdef ENGINE_PlayerController_generated_h
#error "PlayerController.generated.h already included, missing '#pragma once' in PlayerController.h"
#endif
#define ENGINE_PlayerController_generated_h

There are more things worthy to check.

1
2
3
4
5
6
7
8
9
10
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/CodeGenerator.cpp

FString MacroName = SourceFile.GetGeneratedMacroName(ClassData, TEXT("_EVENT_PARMS"));
WriteMacro(OutGeneratedHeaderText, MacroName, UClassMacroContent);
PrologMacroCalls.Logf(TEXT("\t%s\r\n"), *MacroName);

...

GeneratedHeaderText.Log(TEXT("#undef CURRENT_FILE_ID\r\n"));
GeneratedHeaderText.Logf(TEXT("#define CURRENT_FILE_ID %s\r\n\r\n\r\n"), *SourceFile->GetFileId());

That is why the CURRENT_FILE_ID and ..._EVENT_PARMS macros are defined. Furthermore, other codes can be found at FNativeClassHeaderGenerator::FNativeClassHeaderGenerator(const UPackage*, const TSet<FUnrealSourceFile*>&, FClasses&, bool).

So, we have found the relationship of unreal macro and generated header file. But, there is one thing left, the metadata.

Metadata Parser

1
2
3
4
/UnrealEngine/Engine/Source/Runtime/Engine/Classes/GameFramework/PlayerController.h

UCLASS(config=Game, BlueprintType, Blueprintable, meta=(ShortTooltip="A Player Controller is an actor responsible for controlling a Pawn used by the player."))
class ENGINE_API APlayerController : public AController

Back to the start, there are metadata within the macro UCLASS such as config=Game, BlueprintType and meta=.... We are going to check out how they are handled by UnrealEngine.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
/UnrealEngine/Engine/Source/Runtime/CoreUObject/Public/UObject/ObjectMacros.h

// These are used for syntax highlighting and to allow autocomplete hints

namespace UC
{
// valid keywords for the UCLASS macro
enum
{
/// This keyword is used to set the actor group that the class is show in, in the editor.
classGroup,

/// Declares that instances of this class should always have an outer of the specified class. This is inherited by subclasses unless overridden.
Within, /* =OuterClassName */

/// Exposes this class as a type that can be used for variables in blueprints
BlueprintType,

/// Prevents this class from being used for variables in blueprints
NotBlueprintType,

/// Exposes this class as an acceptable base class for creating blueprints. The default is NotBlueprintable, unless inherited otherwise. This is inherited by subclasses.
Blueprintable,

/// Specifies that this class is *NOT* an acceptable base class for creating blueprints. The default is NotBlueprintable, unless inherited otherwise. This is inherited by subclasses.
NotBlueprintable,
...

You can find some enum definitions seems related to the metadata. But, they are for only supporting autocomplete hints such as Intellisense and VisualAssistX. There is another code handling metadata.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/UnrealEngine/Engine/Source/Programs/UnrealHeader/Private/CodeGenerator.cpp

const TArray<FString>& UObjectHeaders =
(CurrentlyProcessing == PublicClassesHeaders) ? Module.PublicUObjectClassesHeaders :
(CurrentlyProcessing == PublicHeaders ) ? Module.PublicUObjectHeaders :
Module.PrivateUObjectHeaders;
...
ParallelFor(UObjectHeaders.Num(), [&](int32 Index)
{
const FString& RawFilename = UObjectHeaders[Index];

#if !PLATFORM_EXCEPTIONS_DISABLED
try
#endif
{
PerformSimplifiedClassParse(Package, *RawFilename, *HeaderFiles[Index], PerHeaderData[Index]);
}
...

This code is for parsing metadata. Any header file in your project is passed through the parsing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
static void SetupUObjectModuleHeader(UHTModuleInfo ModuleInfo, FileItem HeaderFile, SourceFileMetadataCache MetadataCache)
{
// Check to see if we know anything about this file. If we have up-to-date cached information about whether it has
// UObjects or not, we can skip doing a test here.
if (MetadataCache.ContainsReflectionMarkup(HeaderFile))
{
lock(ModuleInfo)
{
bool bFoundHeaderLocation = false;
foreach (DirectoryReference ModuleDirectory in ModuleInfo.ModuleDirectories)
{
if (HeaderFile.Location.IsUnderDirectory(DirectoryReference.Combine(ModuleDirectory, "Classes")))
{
ModuleInfo.PublicUObjectClassesHeaders.Add(HeaderFile);
bFoundHeaderLocation = true;
}
else if (HeaderFile.Location.IsUnderDirectory(DirectoryReference.Combine(ModuleDirectory, "Public")))
{
ModuleInfo.PublicUObjectHeaders.Add(HeaderFile);
bFoundHeaderLocation = true;
}
}
if (!bFoundHeaderLocation)
{
ModuleInfo.PrivateUObjectHeaders.Add(HeaderFile);
}
}
}
}

About all modules, header files in Classes folder are stored at PublicUObjectClassesHeaders and header files in Public folder are stored at PublicUObjectHeaders. Even you have located a header file in other folder, the Unreal Build Tool collects it into PrivateUObjectHeaders.

A screenshot on debugging UBT.

Back to the FBaseParser::ReadSpecifierSetInsideMacro(), let us test with the keyword BlueprintType. How does the BlueprintType keyword parsed ? The UHT parses your header file with tokens. Suppose the input as UCLASS(config=Game, BlueprintType, Blueprintable, meta=(...)).

1
2
3
4
5
6
7
8
9
10
/UnrealEngine/Engine/Source/Programs/UnrealHeaderTool/Private/HeaderParser.cpp

if (Token.Matches(TEXT("UCLASS"), ESearchCase::CaseSensitive))
{
bHaveSeenUClass = true;
bEncounteredNewStyleClass_UnmatchedBrackets = true;
UClass* Class = CompileClassDeclaration(AllClasses);
GStructToSourceLine.Add(Class, MakeTuple(GetCurrentSourceFile()->AsShared(), Token.StartLine));
return true;
}

Due to this code, the left input would be (config=Game, BlueprintType, Blueprintable, meta=(...)). And, the following tokenizing is like below based on FBaseParser::ReadSpecifierSetInsideMacro().

1
2
3
4
5
6
7
8
9
10
(config=Game, BlueprintType, Blueprintable, meta=(...))
-> RequireSymbol(TEXT('('), ErrorMessageGetter);
config=Game, BlueprintType, Blueprintable, meta=(...))
-> GetToken(Specifier); SpecifiersFound.Emplace(Specifier.Identifier);
, BlueprintType, Blueprintable, meta=(...))
-> RequireSymbol(TEXT(','), ErrorMessageGetter);
BlueprintType, Blueprintable, meta=(...))
-> GetToken(Specifier); GetMetadataKeyword(Specifier.Identifier);
, Blueprintable, meta=(...))
...

A screenshot on debugging UHT.

Wrap-Up

1
2
3
4
5
6
7
8
C/CPP (pure) macro
C/CPP code with macro ---(preprocessor)--->
C/CPP code with evaluated code from macro ---(rest of job)---> ...

unreal macro
C/CPP code with unreal macro ---(UHT and UBT)--->
C/CPP code with generated code(+macro) from UHT and UBT ---(preprocessor)--->
C/CPP code with evaluated code from macro ---(rest of job)---> ...

There are so many hidden code for implementing unreal macros, and the macros have complicated relationship with other engine code. Even most part of final code from the macros cannot be evaluated before some preprocessing and compilation. In this perspective, unreal macro such as UCLASS is not a pure C/CPP macro, because unreal macro functions fully only when UHT and UBT must preprocess the macro.

There is no doubt. Any of C/CPP compiler cannot recognize the unreal macro such as UCLASS. Even the Epic Games did not modify the compilers, and did not have to do. They have simply setup some build pipeline satisfying their needs. The program managing their custom build pipeline is the Unreal Build Tool, UBT. Most of jobs for build are done by UBT and UHT. In official document for UHT, these background knowledge is introduced.

1
2
3
4
5
6
UnrealHeaderTool (UHT) is a custom parsing and code-generation tool that supports the UObject system. Code compilation happens in two phases:

1. UHT is invoked, which parses the C++ headers for Unreal-related class metadata and generates custom code to implement the various UObject-related features.
2. The normal C++ compiler is invoked to compile the results.

When compiling, it is possible for either tool to emit errors, so be sure to look carefully.

As they said, the compilation order is the opposite direction of the paragraphs; Expanding UCLASS, Generated Header File and Metadata Parser. The actions for Expanding UCLASS are done by C/CPP compilers(+preprocessors), and the actions for Generated Header File and Metadata Parser are done by UHT. Additionally, actions for Metadata Parser happens early than ones for Generated Header File.

1
2
main function for parsing metadata -> FHeaderParser::ParseHeaders()
main function for generating header file -> FHeaderParser::ExportNativeHeaders()

Let us make a conclusion.

  • The result from unreal macro is hard to evaluate before processing by UBT(+UHT).
  • Some features of UnrealEngine are implemented by auto-generated codes.
  • You should look into the build pipeline of UnrealEngine if need to modify unreal macro things.

Growth of std::vector

Overview

Nowadays, STL is an essential component in almost every CPP project. That is why several questions about STL are asked in a technical interview. Especially, the std::vector is a popular subject. In this post, we gonna check the codes related to std::vector‘s growth, which is a hot topic in STL.
The term “growth” in std::vector means an event to increase a size of instance by some actions. The action would be inserting an element (e.g. push_back()) or tuning its size (e.g. resize()). Some of us say “When the growth happens, its size become twice.”, but some of others say “No, it is exactly 3/2 times.”. Well…both of saying are not wrong. Let us find out why it is.

The environment is below:

  • MSVC
    • Windows 10 2004 OS Build 19041.985
    • MSVC 19.28.29914
  • GCC
    • Ubuntu 18 18.04.4 LTS
    • GCC 7.5.0

The reference is below:

Growth in MSVC

First, let us trace codes about push_back(). Suppose we use a code below:

1
2
3
int Number = 0;
std::vector<int> Instance;
Instance.push_back(Number);

The function below will be called in this code. You can find all of STL codes for MSVC at reference #2.

1
2
3
_CONSTEXPR20_CONTAINER void push_back(const _Ty& _Val) { // insert element at end, provide strong guarantee
emplace_back(_Val);
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
template <class... _Valty>
_CONSTEXPR20_CONTAINER decltype(auto) emplace_back(_Valty&&... _Val) {
// insert by perfectly forwarding into element at end, provide strong guarantee
auto& _My_data = _Mypair._Myval2;
pointer& _Mylast = _My_data._Mylast;
if (_Mylast != _My_data._Myend) {
return _Emplace_back_with_unused_capacity(_STD forward<_Valty>(_Val)...);
}

_Ty& _Result = *_Emplace_reallocate(_Mylast, _STD forward<_Valty>(_Val)...);
#if _HAS_CXX17
return _Result;
#else // ^^^ _HAS_CXX17 ^^^ // vvv !_HAS_CXX17 vvv
(void) _Result;
#endif // _HAS_CXX17
}

As you can see, there is a branch on returning the function. When _Mylast != _My_data._Myend is true, the growth not happens. Because the logic in the _Emplace_back_with_unused_capacity() does not reallocate memory, but reuse unused memory. FYI, values about _Mypair have the relationship like below:

1
_Compressed_pair<_Alty, _Scary_val> _Mypair;
1
2
3
4
5
// https://github.com/microsoft/STL/blob/c12089e489c7b6a3896f5043ed545ac8d1870590/stl/inc/xmemory
template <class _Ty1, class _Ty2, bool = is_empty_v<_Ty1> && !is_final_v<_Ty1>>
class _Compressed_pair final : private _Ty1 { // store a pair of values, deriving from empty first
public:
_Ty2 _Myval2;
1
2
3
4
5
6
7
8
9
10
11
12
// CLASS TEMPLATE vector
template <class _Ty, class _Alloc = allocator<_Ty>>
class vector { // varying size array of values
private:
template <class>
friend class _Vb_val;
friend _Tidy_guard<vector>;

using _Alty = _Rebind_alloc_t<_Alloc, _Ty>;
...
using _Scary_val = _Vector_val<conditional_t<_Is_simple_alloc_v<_Alty>, _Simple_types<_Ty>,
_Vec_iter_types<_Ty, size_type, difference_type, pointer, const_pointer, _Ty&, const _Ty&>>>;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// CLASS TEMPLATE _Vector_val
template <class _Val_types>
class _Vector_val : public _Container_base {
public:
using value_type = typename _Val_types::value_type;
using size_type = typename _Val_types::size_type;
using difference_type = typename _Val_types::difference_type;
using pointer = typename _Val_types::pointer;
using const_pointer = typename _Val_types::const_pointer;
using reference = value_type&;
using const_reference = const value_type&;

_CONSTEXPR20_CONTAINER _Vector_val() noexcept : _Myfirst(), _Mylast(), _Myend() {}

_CONSTEXPR20_CONTAINER _Vector_val(pointer _First, pointer _Last, pointer _End) noexcept
: _Myfirst(_First), _Mylast(_Last), _Myend(_End) {}

_CONSTEXPR20_CONTAINER void _Swap_val(_Vector_val& _Right) noexcept {
this->_Swap_proxy_and_iterators(_Right);
_Swap_adl(_Myfirst, _Right._Myfirst);
_Swap_adl(_Mylast, _Right._Mylast);
_Swap_adl(_Myend, _Right._Myend);
}

_CONSTEXPR20_CONTAINER void _Take_contents(_Vector_val& _Right) noexcept {
this->_Swap_proxy_and_iterators(_Right);
_Myfirst = _Right._Myfirst;
_Mylast = _Right._Mylast;
_Myend = _Right._Myend;

_Right._Myfirst = nullptr;
_Right._Mylast = nullptr;
_Right._Myend = nullptr;
}

pointer _Myfirst; // pointer to beginning of array
pointer _Mylast; // pointer to current end of sequence
pointer _Myend; // pointer to end of array
};

Since the elements are placed in sequential memory address, _Mylast - _Myfirst means “currently used size”.

1
2
3
4
_NODISCARD _CONSTEXPR20_CONTAINER size_type size() const noexcept {
auto& _My_data = _Mypair._Myval2;
return static_cast<size_type>(_My_data._Mylast - _My_data._Myfirst);
}

Similarly, _Myend - _Myfirst means “currently avaiable size”.

1
2
3
4
_NODISCARD _CONSTEXPR20_CONTAINER size_type capacity() const noexcept {
auto& _My_data = _Mypair._Myval2;
return static_cast<size_type>(_My_data._Myend - _My_data._Myfirst);
}

As a result, _Mylast != _My_data._Myend is true when _Mylast < _Myend is true. That is why reallocation not happens. Get back to emplace_back() code. According to those upper reasons, now we need to focus on _Emplace_reallocated() function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
template <class... _Valty>
_CONSTEXPR20_CONTAINER pointer _Emplace_reallocate(const pointer _Whereptr, _Valty&&... _Val) {
// reallocate and insert by perfectly forwarding _Val at _Whereptr
_Alty& _Al = _Getal();
auto& _My_data = _Mypair._Myval2;
pointer& _Myfirst = _My_data._Myfirst;
pointer& _Mylast = _My_data._Mylast;

_STL_INTERNAL_CHECK(_Mylast == _My_data._Myend); // check that we have no unused capacity

const auto _Whereoff = static_cast<size_type>(_Whereptr - _Myfirst);
const auto _Oldsize = static_cast<size_type>(_Mylast - _Myfirst);

if (_Oldsize == max_size()) {
_Xlength();
}

const size_type _Newsize = _Oldsize + 1;
const size_type _Newcapacity = _Calculate_growth(_Newsize);

const pointer _Newvec = _Al.allocate(_Newcapacity);
const pointer _Constructed_last = _Newvec + _Whereoff + 1;
pointer _Constructed_first = _Constructed_last;

_TRY_BEGIN
_Alty_traits::construct(_Al, _Unfancy(_Newvec + _Whereoff), _STD forward<_Valty>(_Val)...);
_Constructed_first = _Newvec + _Whereoff;

if (_Whereptr == _Mylast) { // at back, provide strong guarantee
_Umove_if_noexcept(_Myfirst, _Mylast, _Newvec);
} else { // provide basic guarantee
_Umove(_Myfirst, _Whereptr, _Newvec);
_Constructed_first = _Newvec;
_Umove(_Whereptr, _Mylast, _Newvec + _Whereoff + 1);
}
_CATCH_ALL
_Destroy(_Constructed_first, _Constructed_last);
_Al.deallocate(_Newvec, _Newcapacity);
_RERAISE;
_CATCH_END

_Change_array(_Newvec, _Newsize, _Newcapacity);
return _Newvec + _Whereoff;
}

As you can see the codes, deallocation and reallocation happen. The variable _Newcapacity determines the size of memory will be reallocated. Let us check the function _Calculate_growth().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
_NODISCARD _CONSTEXPR20_CONTAINER size_type max_size() const noexcept {
return (_STD min)(
static_cast<size_type>((numeric_limits<difference_type>::max)()), _Alty_traits::max_size(_Getal()));
}
...
_CONSTEXPR20_CONTAINER size_type _Calculate_growth(const size_type _Newsize) const {
// given _Oldcapacity and _Newsize, calculate geometric growth
const size_type _Oldcapacity = capacity();
const auto _Max = max_size();

if (_Oldcapacity > _Max - _Oldcapacity / 2) {
return _Max; // geometric growth would overflow
}

const size_type _Geometric = _Oldcapacity + _Oldcapacity / 2;

if (_Geometric < _Newsize) {
return _Newsize; // geometric growth would be insufficient
}

return _Geometric; // geometric growth is sufficient
}

There are three return statement in the function.

  • First, when the current available size is bigger than 2/3 times of maximum size.

For instance, a maximum value of int type is +2,147,483,647 and 2/3 times of value is +1,431,655,764.666... ≒ +1,431,655,765. Let us put them in the expression. if (1431655765 > 2147483647 - 1431655765 / 2) will be false, but how about if _Oldcapacity = +1,431,655,766 ? if (1431655766 > 2147483647 - 1431655766 / 2) will be true. In this case, new size will be forced as the maximum size.

  • Second, when the current available size is less than 2.

For instance, when the _Oldcapacity is in {0, 1} the expression const size_type _Geometric = _Oldcapacity + _Oldcapacity / 2; will be the same with _Oldcapacity. In this case, new size will be forced as _Newsize, which is passed by _Oldsize + 1 in _Emplace_reallocate().

_Oldcapacity Calculation
0 0 + 0 / 2 = 0
1 1 + 1 / 2 = 1
  • Third, other cases of the current available size.

The _Geometric will have 3/2 times of _Oldcapacity. That is why the 3/2 times of growth happens in MSVC. And now you understand why new size has to be set by maximum value when the _Oldcapacity is bigger than 2/3 times of maximum size.

The resize() has a similar flow. Let us find out.

1
2
3
4
_CONSTEXPR20_CONTAINER void resize(_CRT_GUARDOVERFLOW const size_type _Newsize) {
// trim or append value-initialized elements, provide strong guarantee
_Resize(_Newsize, _Value_init_tag{});
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
template <class _Ty2>
_CONSTEXPR20_CONTAINER void _Resize(const size_type _Newsize, const _Ty2& _Val) {
// trim or append elements, provide strong guarantee
auto& _My_data = _Mypair._Myval2;
pointer& _Myfirst = _My_data._Myfirst;
pointer& _Mylast = _My_data._Mylast;
const auto _Oldsize = static_cast<size_type>(_Mylast - _Myfirst);
if (_Newsize < _Oldsize) { // trim
const pointer _Newlast = _Myfirst + _Newsize;
_Orphan_range(_Newlast, _Mylast);
_Destroy(_Newlast, _Mylast);
_Mylast = _Newlast;
return;
}

if (_Newsize > _Oldsize) { // append
const auto _Oldcapacity = static_cast<size_type>(_My_data._Myend - _Myfirst);
if (_Newsize > _Oldcapacity) { // reallocate
_Resize_reallocate(_Newsize, _Val);
return;
}

const pointer _Oldlast = _Mylast;
_Mylast = _Ufill(_Oldlast, _Newsize - _Oldsize, _Val);
_Orphan_range(_Oldlast, _Oldlast);
}

// if _Newsize == _Oldsize, do nothing; avoid invalidating iterators
}

The resize() can trim or append available memory. Trimming happens when you call resize() with smaller value than current available size. Appending happens when you call resize() with greater value than current available size. We go to _Resize_reallocate().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
template <class _Ty2>
_CONSTEXPR20_CONTAINER void _Resize_reallocate(const size_type _Newsize, const _Ty2& _Val) {
if (_Newsize > max_size()) {
_Xlength();
}

auto& _My_data = _Mypair._Myval2;
pointer& _Myfirst = _My_data._Myfirst;
pointer& _Mylast = _My_data._Mylast;

const auto _Oldsize = static_cast<size_type>(_Mylast - _Myfirst);
const size_type _Newcapacity = _Calculate_growth(_Newsize);

const pointer _Newvec = _Getal().allocate(_Newcapacity);
const pointer _Appended_first = _Newvec + _Oldsize;
pointer _Appended_last = _Appended_first;

_TRY_BEGIN
_Appended_last = _Ufill(_Appended_first, _Newsize - _Oldsize, _Val);
_Umove_if_noexcept(_Myfirst, _Mylast, _Newvec);
_CATCH_ALL
_Destroy(_Appended_first, _Appended_last);
_Getal().deallocate(_Newvec, _Newcapacity);
_RERAISE;
_CATCH_END

_Change_array(_Newvec, _Newsize, _Newcapacity);
}

Oh, Hi. We meet again. It is him, the _Calculate_growth(). Now we know the resize() has a similar logic.

Growth in GCC

First of all, let us find push_back() in GCC. Suppose we use the example written at MSVC part. You can find the code at reference #3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// [23.2.4.3] modifiers
/**
* @brief Add data to the end of the %vector.
* @param __x Data to be added.
*
* This is a typical stack operation. The function creates an
* element at the end of the %vector and assigns the given data
* to it. Due to the nature of a %vector this operation can be
* done in constant time if the %vector has preallocated space
* available.
*/
void
push_back(const value_type& __x)
{
if (this->_M_impl._M_finish != this->_M_impl._M_end_of_storage)
{
_GLIBCXX_ASAN_ANNOTATE_GROW(1);
_Alloc_traits::construct(this->_M_impl, this->_M_impl._M_finish,
__x);
++this->_M_impl._M_finish;
_GLIBCXX_ASAN_ANNOTATE_GREW(1);
}
else
_M_realloc_insert(end(), __x);
}

Here are two cases. First, when an available memory exists. Second, otherwise.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
struct _Vector_impl_data
{
pointer _M_start;
pointer _M_finish;
pointer _M_end_of_storage;

_Vector_impl_data() _GLIBCXX_NOEXCEPT
: _M_start(), _M_finish(), _M_end_of_storage()
{ }

#if __cplusplus >= 201103L
_Vector_impl_data(_Vector_impl_data&& __x) noexcept
: _M_start(__x._M_start), _M_finish(__x._M_finish),
_M_end_of_storage(__x._M_end_of_storage)
{ __x._M_start = __x._M_finish = __x._M_end_of_storage = pointer(); }
#endif

void
_M_copy_data(_Vector_impl_data const& __x) _GLIBCXX_NOEXCEPT
{
_M_start = __x._M_start;
_M_finish = __x._M_finish;
_M_end_of_storage = __x._M_end_of_storage;
}

void
_M_swap_data(_Vector_impl_data& __x) _GLIBCXX_NOEXCEPT
{
// Do not use std::swap(_M_start, __x._M_start), etc as it loses
// information used by TBAA.
_Vector_impl_data __tmp;
__tmp._M_copy_data(*this);
_M_copy_data(__x);
__x._M_copy_data(__tmp);
}
};
...
_Vector_impl _M_impl;

std::vector in GCC has internal indicators like std::vector in MSVC. So we should focus on _M_realloc_insert() function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
#if __cplusplus >= 201103L
template<typename _Tp, typename _Alloc>
template<typename... _Args>
void
vector<_Tp, _Alloc>::
_M_realloc_insert(iterator __position, _Args&&... __args)
#else
template<typename _Tp, typename _Alloc>
void
vector<_Tp, _Alloc>::
_M_realloc_insert(iterator __position, const _Tp& __x)
#endif
{
const size_type __len =
_M_check_len(size_type(1), "vector::_M_realloc_insert");
pointer __old_start = this->_M_impl._M_start;
pointer __old_finish = this->_M_impl._M_finish;
const size_type __elems_before = __position - begin();
pointer __new_start(this->_M_allocate(__len));
pointer __new_finish(__new_start);
__try
{
// The order of the three operations is dictated by the C++11
// case, where the moves could alter a new element belonging
// to the existing vector. This is an issue only for callers
// taking the element by lvalue ref (see last bullet of C++11
// [res.on.arguments]).
_Alloc_traits::construct(this->_M_impl,
__new_start + __elems_before,
#if __cplusplus >= 201103L
std::forward<_Args>(__args)...);
#else
__x);
#endif
__new_finish = pointer();

#if __cplusplus >= 201103L
if _GLIBCXX17_CONSTEXPR (_S_use_relocate())
{
__new_finish = _S_relocate(__old_start, __position.base(),
__new_start, _M_get_Tp_allocator());

++__new_finish;

__new_finish = _S_relocate(__position.base(), __old_finish,
__new_finish, _M_get_Tp_allocator());
}
else
#endif
{
__new_finish
= std::__uninitialized_move_if_noexcept_a
(__old_start, __position.base(),
__new_start, _M_get_Tp_allocator());

++__new_finish;

__new_finish
= std::__uninitialized_move_if_noexcept_a
(__position.base(), __old_finish,
__new_finish, _M_get_Tp_allocator());
}
}
__catch(...)
{
if (!__new_finish)
_Alloc_traits::destroy(this->_M_impl,
__new_start + __elems_before);
else
std::_Destroy(__new_start, __new_finish, _M_get_Tp_allocator());
_M_deallocate(__new_start, __len);
__throw_exception_again;
}
#if __cplusplus >= 201103L
if _GLIBCXX17_CONSTEXPR (!_S_use_relocate())
#endif
std::_Destroy(__old_start, __old_finish, _M_get_Tp_allocator());
_GLIBCXX_ASAN_ANNOTATE_REINIT;
_M_deallocate(__old_start,
this->_M_impl._M_end_of_storage - __old_start);
this->_M_impl._M_start = __new_start;
this->_M_impl._M_finish = __new_finish;
this->_M_impl._M_end_of_storage = __new_start + __len;
}

Hoo, it is too long. We do not have to look into whole code, but the variable __len. The variable is used for reallocation. And it is set by _M_check_len().

1
2
3
4
5
6
7
8
9
10
// Called by _M_fill_insert, _M_insert_aux etc.
size_type
_M_check_len(size_type __n, const char* __s) const
{
if (max_size() - size() < __n)
__throw_length_error(__N(__s));

const size_type __len = size() + (std::max)(size(), __n);
return (__len < size() || __len > max_size()) ? max_size() : __len;
}

The code throw an error when current size is the same with maximum size because the function was called as _M_check_len(size_type(1), ...). Otherwise, new size will be set by 2 times of current size. Except for when current size is 0.

Current size Calculation
0 1 = 0 + max(0, 1)
1 2 = 1 + max(1, 1)
2 4 = 2 + max(2, 1)

And, returns maximum size when underflow or overflow happens. Otherwise, returns new size calculated as 2 times of current size.

Next, check the resize() in GCC.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/**
* @brief Resizes the %vector to the specified number of elements.
* @param __new_size Number of elements the %vector should contain.
*
* This function will %resize the %vector to the specified
* number of elements. If the number is smaller than the
* %vector's current size the %vector is truncated, otherwise
* default constructed elements are appended.
*/
void
resize(size_type __new_size)
{
if (__new_size > size())
_M_default_append(__new_size - size());
else if (__new_size < size())
_M_erase_at_end(this->_M_impl._M_start + __new_size);
}

We can see the resize() in GCC also do trimming and appending. (Interestingly, nothing happens when __new_size is equal to current size.) So, we should focus on _M_default_append() function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
template<typename _Tp, typename _Alloc>
void
vector<_Tp, _Alloc>::
_M_default_append(size_type __n)
{
if (__n != 0)
{
const size_type __size = size();
size_type __navail = size_type(this->_M_impl._M_end_of_storage
- this->_M_impl._M_finish);

if (__size > max_size() || __navail > max_size() - __size)
__builtin_unreachable();

if (__navail >= __n)
{
_GLIBCXX_ASAN_ANNOTATE_GROW(__n);
this->_M_impl._M_finish =
std::__uninitialized_default_n_a(this->_M_impl._M_finish,
__n, _M_get_Tp_allocator());
_GLIBCXX_ASAN_ANNOTATE_GREW(__n);
}
else
{
const size_type __len =
_M_check_len(__n, "vector::_M_default_append");
pointer __new_start(this->_M_allocate(__len));
if _GLIBCXX17_CONSTEXPR (_S_use_relocate())
{
__try
{
std::__uninitialized_default_n_a(__new_start + __size,
__n, _M_get_Tp_allocator());
}
__catch(...)
{
_M_deallocate(__new_start, __len);
__throw_exception_again;
}
_S_relocate(this->_M_impl._M_start, this->_M_impl._M_finish,
__new_start, _M_get_Tp_allocator());
}
else
{
pointer __destroy_from = pointer();
__try
{
std::__uninitialized_default_n_a(__new_start + __size,
__n, _M_get_Tp_allocator());
__destroy_from = __new_start + __size;
std::__uninitialized_move_if_noexcept_a(
this->_M_impl._M_start, this->_M_impl._M_finish,
__new_start, _M_get_Tp_allocator());
}
__catch(...)
{
if (__destroy_from)
std::_Destroy(__destroy_from, __destroy_from + __n,
_M_get_Tp_allocator());
_M_deallocate(__new_start, __len);
__throw_exception_again;
}
std::_Destroy(this->_M_impl._M_start, this->_M_impl._M_finish,
_M_get_Tp_allocator());
}
_GLIBCXX_ASAN_ANNOTATE_REINIT;
_M_deallocate(this->_M_impl._M_start,
this->_M_impl._M_end_of_storage
- this->_M_impl._M_start);
this->_M_impl._M_start = __new_start;
this->_M_impl._M_finish = __new_start + __size + __n;
this->_M_impl._M_end_of_storage = __new_start + __len;
}
}
}

It is long one, too. What is __navail ? It seems meaning of Number of AVAILable memory, not the Not AVAILable memory. So, we can see the memory is reused when if (__navail >= __n) is true. Otherwise, reallocation happens. Oh, Hi. We meet _M_check_len() again. Then, new size will be 2 times of current size.

Wrap-up

Common

  • Try to recycle memory as possible as can. (e.g. reuse available memory in push_back() logic.)
  • Care about underflow and overflow.
  • Have internal indicators for {First, Current, End}
    • Currently allocated size = End - First
    • Currently used size = Current - First
    • Currently available size = End - Current

MSVC

  • Growth happens with 3/2 times of amount. (in normal case)

GCC

  • Growth happens with 2 times of amount. (in normal case)

Differences between Build.cs and Target.cs in UnrealEngine

UnrealBuildTool.ModuleRules

UnrealEngine provides its own module system, which is absolutely different with CPP 20 Module. The class UnrealBuildTool.ModuleRules is for the module system and it is written by [ModuleName].Build.cs. You can decide what to include for creating output files (= DLL). For example, ShaderCompileWorker project has the module rules below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
/Engine/Source/Programs/ShaderCompileWorker/ShaderCompileWorker.Build.cs

public class ShaderCompileWorker : ModuleRules
{
public ShaderCompileWorker(ReadOnlyTargetRules Target) : base(Target)
{
PrivateDependencyModuleNames.AddRange(
new string[] {
"Core",
"Projects",
"RenderCore",
"SandboxFile",
"TargetPlatform",
"ApplicationCore",
"TraceLog",
"ShaderCompilerCommon"
});

if (Target.Platform == UnrealTargetPlatform.Linux)
{
PrivateDependencyModuleNames.AddRange(
new string[] {
"NetworkFile",
"PakFile",
"StreamingFile",
});
}

PrivateIncludePathModuleNames.AddRange(
new string[] {
"Launch",
"TargetPlatform",
});

PrivateIncludePaths.Add("Runtime/Launch/Private"); // For LaunchEngineLoop.cpp include

// Include D3D compiler binaries
string EngineDir = Path.GetFullPath(Target.RelativeEnginePath);

if (Target.Platform == UnrealTargetPlatform.Win32)
{
RuntimeDependencies.Add(EngineDir + "Binaries/ThirdParty/Windows/DirectX/x86/d3dcompiler_47.dll");
}
else if (Target.Platform == UnrealTargetPlatform.Win64)
{
RuntimeDependencies.Add(EngineDir + "Binaries/ThirdParty/Windows/DirectX/x64/d3dcompiler_47.dll");
}
}
}

There are several libraries such as Core, Projects, RenderCore and so on. We can also find them in Binaries folder like below:
(FYI, the ShaderCompileWorker.exe is created with ShaderCompileWorker.Target.cs not the ShaderCompileWorker.Build.cs.)

In other words, output files are created with the name containing its module when you add corresponding libraries to module rules (ex: [TargetName]-[ModuleName]-[ConfigurationName].dll). Additionally, UnrealEngine re-uses them as possible. Suppose your project need some libraries already included on engine side. In this situation, UnrealEngine does not create output files for the duplicated libraries included in your project. Instead of that, UnrealEngine leaves some meta file describes what the project included.

1
2
3
4
5
6
7
8
9
10
11
12
13
// Copyright Epic Games, Inc. All Rights Reserved.

using UnrealBuildTool;

public class ThirdPerson_4_25 : ModuleRules
{
public ThirdPerson_4_25(ReadOnlyTargetRules Target) : base(Target)
{
PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;

PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "HeadMountedDisplay" });
}
}

This is a generated module rules based on third person template. This module rules contains CoreUObject library but we cannot find it in Binaries folder. Let us see the ThirdPerson_4_25Editor.target file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
/Project/Binaries/Win64/ThirdPerson_4_25Editor.target

...
"BuildProducts": [
...
{
"Path": "$(EngineDir)/Binaries/Win64/UE4Editor-CoreUObject.dll",
"Type": "DynamicLibrary"
},
{
"Path": "$(EngineDir)/Binaries/Win64/UE4Editor-CoreUObject.pdb",
"Type": "SymbolFile"
},
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
/Engine/Source/Programs/UnrealBuildTool/System/TargetReceipt.cs

...
/// <summary>
/// Write the receipt to disk.
/// </summary>
/// <param name="Location">Output filename</param>
/// <param name="EngineDir">Engine directory for expanded paths</param>
public void Write(FileReference Location, DirectoryReference EngineDir)
{
...
Writer.WriteArrayStart("BuildProducts");
foreach (BuildProduct BuildProduct in BuildProducts)
{
Writer.WriteObjectStart();
Writer.WriteValue("Path", InsertPathVariables(BuildProduct.Path, EngineDir, ProjectDir));
Writer.WriteValue("Type", BuildProduct.Type.ToString());
Writer.WriteObjectEnd();
}
Writer.WriteArrayEnd();
...

For more details about the module system, visit reference #1.

UnrealBuildTool.TargetRules

UnrealEngine provides its own target system, which makes you can create an executable. There are various target configurations such as Editor, Client and Server. The class UnrealBuildTool.TargetRules is for the target system and it is written by [TargetName].Target.cs. You can decide which modules to include for a certain target. For example, UE4 project has the target rules for editor below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// Copyright Epic Games, Inc. All Rights Reserved.

using UnrealBuildTool;
using System.Collections.Generic;

public class UE4EditorTarget : TargetRules
{
public UE4EditorTarget( TargetInfo Target ) : base(Target)
{
Type = TargetType.Editor;
BuildEnvironment = TargetBuildEnvironment.Shared;
bBuildAllModules = true;
ExtraModuleNames.Add("UE4Game");
}
}

1
ExtraModuleNames.Add("UE4Game");

The target rules specifies UE4Game module included, which is located in Engine/Source/Runtime/UE4Game. So, output files for editor are consist of modules in UE4Game.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// Copyright Epic Games, Inc. All Rights Reserved.

using UnrealBuildTool;

public class UE4Game : ModuleRules
{
public UE4Game(ReadOnlyTargetRules Target) : base(Target)
{
PrivateDependencyModuleNames.Add("Core");

if (Target.Platform == UnrealTargetPlatform.IOS || Target.Platform == UnrealTargetPlatform.TVOS)
{
PrivateDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine" });
if (Target.Platform == UnrealTargetPlatform.IOS)
{
DynamicallyLoadedModuleNames.Add("IOSAdvertising");
}
}
else if (Target.Platform == UnrealTargetPlatform.Android)
{
PrivateDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine" });
DynamicallyLoadedModuleNames.Add("AndroidAdvertising");
}
}
}

Yes, they are. :)

1
Type = TargetType.Editor;

UnrealBuildTool.TargetRules has a field Type.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
/Engine/Source/Programs/UnrealBuildTool/Configuration/TargetRules.cs

...csharp
/// <summary>
/// TargetRules is a data structure that contains the rules for defining a target (application/executable)
/// </summary>
public abstract partial class TargetRules
{
...
/// <summary>
/// The type of target.
/// </summary>
public global::UnrealBuildTool.TargetType Type = global::UnrealBuildTool.TargetType.Game;
...

The field is used for branching target-specific features such as build configuration. For example, UnrealEngine manages target configurations as enum.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
/Engine/Source/Programs/UnrealBuildTool/Configuration/UEBuildTarget.cs

...
/// <summary>
/// The type of configuration a target can be built for
/// </summary>
public enum UnrealTargetConfiguration
{
/// <summary>
/// Unknown
/// </summary>
Unknown,

/// <summary>
/// Debug configuration
/// </summary>
Debug,

/// <summary>
/// DebugGame configuration; equivalent to development, but with optimization disabled for game modules
/// </summary>
DebugGame,

/// <summary>
/// Development configuration
/// </summary>
Development,

/// <summary>
/// Shipping configuration
/// </summary>
Shipping,

/// <summary>
/// Test configuration
/// </summary>
Test,
}
...

Unlike others, the editor target support only 3 types of target configuration. Debug, DebugGame and Development.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
/Engine/Source/Programs/UnrealBuildTool/Configuration/TargetRules.cs

...
/// <summary>
/// Gets a list of configurations that this target supports
/// </summary>
/// <returns>Array of configurations that the target supports</returns>
internal UnrealTargetConfiguration[] GetSupportedConfigurations()
{
// Otherwise take the SupportedConfigurationsAttribute from the first type in the inheritance chain that supports it
for (Type CurrentType = GetType(); CurrentType != null; CurrentType = CurrentType.BaseType)
{
object[] Attributes = CurrentType.GetCustomAttributes(typeof(SupportedConfigurationsAttribute), false);
if (Attributes.Length > 0)
{
return Attributes.OfType<SupportedConfigurationsAttribute>().SelectMany(x => x.Configurations).Distinct().ToArray();
}
}

// Otherwise, get the default for the target type
if (Type == TargetType.Editor)
{
return new[] { UnrealTargetConfiguration.Debug, UnrealTargetConfiguration.DebugGame, UnrealTargetConfiguration.Development };
}
else
{
return ((UnrealTargetConfiguration[])Enum.GetValues(typeof(UnrealTargetConfiguration))).Where(x => x != UnrealTargetConfiguration.Unknown).ToArray();
}
}
...

For more details, visit reference #2.

[ModuleName].Build.cs

Each module has its own Build.cs file. For example, a [ProjectName].Build.cs will be generated when you create new project with cpp enabled. Because UnrealEngine makes a default module that has the same name with project. (Exactly, Build.cs and Target.cs files are copied from template in general cases.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
/Engine/Source/Editor/GameProjectGeneration/Private/GameProjectUtils.cpp

bool GameProjectUtils::CreateProject(const FProjectInformation& InProjectInfo, FText& OutFailReason, FText& OutFailLog, TArray<FString>* OutCreatedFiles)
{
if ( !IsValidProjectFileForCreation(InProjectInfo.ProjectFilename, OutFailReason) )
{
return false;
}

FScopedSlowTask SlowTask(0, LOCTEXT( "CreatingProjectStatus", "Creating project..." ));
SlowTask.MakeDialog();

TOptional<FGuid> ProjectID;
FString TemplateName;
if ( InProjectInfo.TemplateFile.IsEmpty() )
{
ProjectID = GenerateProjectFromScratch(InProjectInfo, OutFailReason, OutFailLog);
TemplateName = InProjectInfo.bShouldGenerateCode ? TEXT("Basic Code") : TEXT("Blank");
}
else
{
ProjectID = CreateProjectFromTemplate(InProjectInfo, OutFailReason, OutFailLog, OutCreatedFiles);
TemplateName = FPaths::GetBaseFilename(InProjectInfo.TemplateFile);
}
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
/Engine/Source/Editor/GameProjectGeneration/Private/GameProjectUtils.cpp

TOptional<FGuid> GameProjectUtils::CreateProjectFromTemplate(const FProjectInformation& InProjectInfo, FText& OutFailReason, FText& OutFailLog, TArray<FString>* OutCreatedFiles)
{
...

// Discover and copy all files in the src folder to the destination, excluding a few files and folders
TArray<FString> FilesToCopy;
TArray<FString> FilesThatNeedContentsReplaced;
TMap<FString, FString> ClassRenames;
IFileManager::Get().FindFilesRecursive(FilesToCopy, *SrcFolder, TEXT("*"), /*Files=*/true, /*Directories=*/false);

...

// Perform the copy
const FString DestFilename = DestFolder / DestFileSubpathWithoutFilename + DestBaseFilename + TEXT(".") + FileExtension;
if ( IFileManager::Get().Copy(*DestFilename, *SrcFilename) == COPY_OK )
{
CreatedFiles.Add(DestFilename);

if ( ReplacementsInFilesExtensions.Contains(FileExtension) )
{
FilesThatNeedContentsReplaced.Add(DestFilename);
}

// Allow project template to extract class renames from this file copy
if (FPaths::GetBaseFilename(SrcFilename) != FPaths::GetBaseFilename(DestFilename)
&& TemplateDefs->IsClassRename(DestFilename, SrcFilename, FileExtension))
{
// Looks like a UObject file!
ClassRenames.Add(FPaths::GetBaseFilename(SrcFilename), FPaths::GetBaseFilename(DestFilename));
}
}
...


WHEN YOU CREATE A PROJECT ITS NAME OF ThridPerson_4_25 FROM THIRD PERSON TEMPLATE

Saying that again, [ModuleName].Build.cs defines the dependencies for building its module. So, every module must have its own [ModuleName].Build.cs file and every module has its own [ModuleName].Build.cs will generate a DLL when you build the project.

Module generation can be done by manipulating some CSharp scripts (Build.cs and Target.cs files). You can find how at https://www.ue4community.wiki/creating-cpp-module-oshdsg2t.

[TargetName].Target.cs

While every module must have a Build.cs file, but every module do not have to have a Target.cs file. Some modules have only Build.cs file. It means the modules should be used for library not a standalone. The AIModule is a good example. The module has only Build.cs as it is written for providing a support to make AI.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
/Engine/Source/Editor/UnrealEd/UnrealEd.Build.cs

...
PublicIncludePathModuleNames.AddRange(
new string[] {
"AssetRegistry",
"AssetTagsEditor",
"CollectionManager",
"BlueprintGraph",
"AddContentDialog",
"MeshUtilities",
"AssetTools",
"KismetCompiler",
"NavigationSystem",
"GameplayTasks",
"AIModule",
"Engine",
"SourceControl",
}
);
...
1
2
3
4
5
6
7
8
9
10
11
12
/Engine/Source/Runtime/Engine/Engine.Build.cs

...
if (Target.bBuildEditor == true)
{
PublicDependencyModuleNames.AddRange(
new string[] {
"UnrealEd",
"Kismet"
}
); // @todo api: Only public because of WITH_EDITOR and UNREALED_API
...
1
2
3
4
5
/Engine/Source/Developer/TargetPlatform/TargetPlatform.Build.cs

...
PrivateIncludePathModuleNames.Add("Engine");
...
1
2
3
4
5
6
7
8
9
10
11
12
13
/Engine/Source/Runtime/Core/Core.Build.cs

...
PrivateIncludePathModuleNames.AddRange(
new string[] {
"TargetPlatform",
"DerivedDataCache",
"InputDevice",
"Analytics",
"RHI"
}
);
...
1
2
3
4
5
/Engine/Source/Runtime/UE4Game/UE4Game.Build.cs

...
PrivateDependencyModuleNames.Add("Core");
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/Engine/Source/UE4Editor.Target.cs

// Copyright Epic Games, Inc. All Rights Reserved.

using UnrealBuildTool;
using System.Collections.Generic;

public class UE4EditorTarget : TargetRules
{
public UE4EditorTarget( TargetInfo Target ) : base(Target)
{
Type = TargetType.Editor;
BuildEnvironment = TargetBuildEnvironment.Shared;
bBuildAllModules = true;
ExtraModuleNames.Add("UE4Game");
}
}

Sometimes, some modules should not be included on certain target configurations. For instance, only editor features should not be included in client or server target configuration. In the need, we can branch for ease like below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Engine/Source/Developer/SlateReflector/SlateReflector.Build.cs

...
// Editor builds include SessionServices to populate the remote target drop-down for remote widget snapshots
if (Target.Type == TargetType.Editor)
{
PublicDefinitions.Add("SLATE_REFLECTOR_HAS_SESSION_SERVICES=1");

PrivateDependencyModuleNames.AddRange(
new string[] {
"PropertyEditor",
}
);

PrivateIncludePathModuleNames.AddRange(
new string[] {
"SessionServices",
}
);

DynamicallyLoadedModuleNames.AddRange(
new string[] {
"SessionServices",
}
);
}
else
{
PublicDefinitions.Add("SLATE_REFLECTOR_HAS_SESSION_SERVICES=0");
}
...

We can use Widget Relfector only at editor target configuration. As we see, non-editor target will not contain the reflector feature. One more step, we can force to block generating projects by throwing an exception like this.

1
2
3
4
5
6
7
8
9
10
11
12
/Engine/Source/Editor/UnrealEd/UnrealEd.Build.cs

...
public class UnrealEd : ModuleRules
{
public UnrealEd(ReadOnlyTargetRules Target) : base(Target)
{
if(Target.Type != TargetType.Editor)
{
throw new BuildException("Unable to instantiate UnrealEd module for non-editor targets.");
}
...

Wrap-Up

Build.cs and Target.cs are different with each other on why they are used.

  • Build.cs
    • defines how the module should be created.
    • type of module output file is usually .dll.
    • every module must have its own [ModuleName].Build.cs.
    • there can be duplicated output files if the module is included in multiple Target.cs files or if build configurations are different on each other.
      • ex: Target1-ModuleA.dll, Target2-ModuleA.dll, Target2-ModuleA-Win64-Debug.dll and etc.
  • Target.cs
    • defines how the executable should be created.
    • type of target output file is usually .exe.
    • user can write [TargetName].Target.cs and modules can be included selectively.
    • there can be duplicated output files if build configurations are different on each other.
      • ex: UE4Editor.exe, UE4Editor-Win64-Debug.exe and etc.

Logically, a target is above one than a module.

Natvis in UnrealEngine

NATive type VISualization

According to reference #1, the name of Natvis framework means visualization of native types. It can help your debugging with more plentiful visibility, and sometimes support mutiple environments that have a different size on the same data type. Suppose you want to make a string class storing its text as UTF-32 or something custom format, it should not be displayed well because it is not kind of ASCII. Though, do you want to see the data (in this case, the text) at Watch or Local viewport ? Then you should implement your own xml for custom Natvis visualization.

SCREENSHOT WITHOUT CUSTOM NATVIS

SCREENSHOT WITH CUSTOM NATVIS

Basic syntax and usage

The custom visualizer has XML syntax and it would be comfortable than a sole programming language. Just create a file of any name with .natvis extension and locate at the directory Documents/Visual Studio 2019/Visualizers. The Visual Studio IDE will find all natvis files at there and parse them. At first after you create the file, you need the tag AutoVisualizer.

1
2
<AutoVisualizer>
</AutoVisualizer>

But you may meet the error like below with our current visualizer. (You should turn on an option for showing errors related to Natvis. Manipulate the option at Tools/Options/Debugging/Output Window/General OutputSettings/Natvisdiagnostic messages. I recommend you to set the level as Error.)

1
2
Natvis: ...\Documents\Visual Studio 2019\Visualizers\Example.natvis(1,2): Fatal error:
Expected element with namespace 'http://schemas.microsoft.com/vstudio/debugger/natvis/2010'.

So you should specify the schemas. Like this.

1
2
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
</AutoVisualizer>

And, the AutoVisualizer tag can have a child tag such as Type. The Type tag must have an attribute Name. Name can be set as the name of type. For example, you should type SomeClass at the attribute when you created a type SomeClass.

1
2
3
4
5
6
7
8
9
10
11
12
13
class SomeClass
{
public:
int ID = 0;
};

int main()
{
SomeClass Some01;
Some01.ID = 100;

return 0;
}
1
2
3
4
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
<Type Name="SomeClass">
</Type>
</AutoVisualizer>

The Type tag can have a child tag such DisplayString/Expand. The DisplayString tag can be used for displaying a string at the debugging window like below.

1
2
3
4
5
6
7
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
<Type Name="SomeClass">
<DisplayString>
This is my class
</DisplayString>
</Type>
</AutoVisualizer>

You can get the value of member variable. Brace the member variable as {}.

1
2
3
4
5
6
7
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
<Type Name = "SomeClass">
<DisplayString>
My ID is {ID}
</DisplayString>
</Type>
</AutoVisualizer>

With the Expand tag, you can customize the expanded view. The Item tags consist of the list. If you customize the expanded view as Expand tag, automatically Raw View item created, which was the original expanded view. You can decorate each line of list in expanded view. The specifier sb and x are respectively meaning “Display the string without quotation marks” and “Display the integer with hexa-decimal format”. For more details, visit reference #2.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
<Type Name = "SomeClass">
<DisplayString>
My ID is {ID}
</DisplayString>
<Expand>
<Item Name="Description">
"Natvis is awesome", sb
</Item>
<Item Name="ID">
ID, x
</Item>
</Expand>
</Type>
</AutoVisualizer>

Example in UnrealEngine

Let us find an example in UnrealEngine one. You can find UE4.natvis if you installed UnrealEngine at your local system. Mostly, the UE4.natvis located in Engine/Extras/VisualStudioDebugging/UE4.natvis.

1
2
3
4
5
6
7
8
9
10
11
12
13
<?xml version="1.0" encoding="utf-8"?>
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
...
<!-- FString visualizer -->
<Type Name="FString">
<DisplayString Condition="Data.ArrayNum == 0">Empty</DisplayString>
<DisplayString Condition="Data.ArrayNum &lt; 0">Invalid</DisplayString>
<DisplayString Condition="Data.ArrayMax &lt; Data.ArrayNum">Invalid</DisplayString>
<DisplayString Condition="Data.ArrayMax &gt;= Data.ArrayNum">{Data.AllocatorInstance.Data,su}</DisplayString>
<StringView Condition="Data.ArrayMax &gt;= Data.ArrayNum">Data.AllocatorInstance.Data,su</StringView>
</Type>
...
</AutoVisualizer>

First of all, the FString welcomes us. Have a look for why the FString visualizer has been made like this. ( &lt; and &gt; things are escaped characters in xml. For more details, visit reference #3. )

Put a breakpoint at where the FString is initialized. Before initialization, we can see Invalid at the debugging window.

Expand the items. We can see the ArrayNum has a negative value. The condition Data.ArrayNum < 0 is satisfied and Invalid would be shown.

After initialization, we can see the string very well. In this case, the condition Data.ArrayMax >= Data.ArrayNum is satisfied and L"ABC" would be shown. Why does the string look like L"..." ? Because of the format specifier su. Check the reference #2 again.

1
2
3
4
5
6
7
8
9
10
11
12
Engine/Source/Runtime/Core/Public/Containers/UnrealString.h

class CORE_API FString
{
private:
friend struct TContainerTraits<FString>;

/** Array holding the character data */
typedef TArray<TCHAR> DataType;
DataType Data;
...
};

The FString stores its string with TArray<TCHAR>. So we could see the ArrayNum or ArrayMax things at the FString visualizer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Engine/Source/Runtime/Core/Public/Containers/Array.h

template<typename InElementType, typename InAllocator>
class TArray
{
template <typename OtherInElementType, typename OtherAllocator>
friend class TArray;

public:
typedef typename InAllocator::SizeType SizeType;
typedef InElementType ElementType;
typedef InAllocator Allocator;

...

protected:

template<typename ElementType, typename Allocator>
friend class TIndirectArray;

ElementAllocatorType AllocatorInstance;
SizeType ArrayNum;
SizeType ArrayMax;
1
2
3
4
5
6
7
8
9
10
11
Engine/Source/Runtime/Core/Public/Containers/ContainersFwd.h

template<int IndexSize> class TSizedDefaultAllocator;
using FDefaultAllocator = TSizedDefaultAllocator<32>;
using FDefaultAllocator64 = TSizedDefaultAllocator<64>;
class FDefaultSetAllocator;

class FString;

template<typename T, typename Allocator = FDefaultAllocator> class TArray;
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Engine/Source/Runtime/Core/Public/Containers/ContainerAllocationPolicies.h

...
template <int IndexSize>
struct TBitsToSizeType
{
static_assert(IndexSize, "Unsupported allocator index size.");
};

template <> struct TBitsToSizeType<8> { using Type = int8; };
template <> struct TBitsToSizeType<16> { using Type = int16; };
template <> struct TBitsToSizeType<32> { using Type = int32; };
template <> struct TBitsToSizeType<64> { using Type = int64; };

/** The indirect allocation policy always allocates the elements indirectly. */
template <int IndexSize>
class TSizedHeapAllocator
{
public:
using SizeType = typename TBitsToSizeType<IndexSize>::Type;
...
};

...
template <int IndexSize> class TSizedDefaultAllocator : public TSizedHeapAllocator<IndexSize> { public: typedef TSizedHeapAllocator<IndexSize> Typedef; };
...

And you can find the type of ArrayNum and ArrayMax is int32 with this flow.

Sometimes, the UE4.natvis gives us a hint for understanding the complicated engine code. Even someday you may need to customize UE4.natvis for special case while supporting various platforms. It would be also good to learn Natvis if you mostly use Visual Studio IDE. Read premade ones and write your ones. :)

Retrospection 2020

overview

Exactly one year passed after I wrote the retrospection of 2019. (https://baemincheon.github.io/2019/12/24/retrospection-2019/) Feeling that time goes so fast as I look back lots of things behind. From the god damn COVID-19 to the remote work during quarantine, we got so many things to say. But, the old year (2019) is still the hardest year to me. This year would be second one as I believe myself that I have taken plenty of breaks in this year.

Though the main topics of retrospection are related to worldwide crisis, I will try not to talk about it as possible. Now it is some kinda routine issue and I want to care about personal events. In addition to, other people are already saying or sharing the contents about it. I just hope this crisis ends as soon as possible and get back into old routines.

play

This year, I played games most of time with Nintendo Switch. Especially, Splatoon 2 and Ring Fit Adventure were the ones that I liked. Splatoon 2 is not such a fresh game (released at 2017/07/21), so it was hard to enjoy multi-play contents due to other players much more skillful than me. But, its single-play contents were quite good and well structured. Therefore, I also purchased a DLC for another single-play contents. I played Splatoon 2 about 100 hours and am pleased with the game quality. The game is not kinda “GOD GAME” thing, but I am sure it has worth to play at least once. For me, an enthusiast of FPS game, unusual concepts of this game looks nice.

I really like the Ring Fit Adventure. It has been so difficult to exercise funny that I used to stop exercising several times. You know, even such a strong will sustains less than few weeks or months. But, I am still keeping exercising for more than 6 months with the game. Thanks to its gamification, I seldom lost interest on exercising and sometimes got motive, too. I played Ring Fit Adventure about 150 hours and am pleased with the game quality. I am sure I can recommend this game if you have Nintendo Switch. And there may be only this option in these days as fitness clubs closed. Home training is not optional but required.

Oh, I almost forgot it. I played the game 3000th Duel, too. It is a game released at PC(Steam) and Nintendo Switch. The game is said as one of Metroidvania things, because of its contents and mood. It was not easy one but there was the balance between feeling fulfilled and bearing hardship. I played 3000th Duel about 40 hours including DLC part. Though It does not have an easy mode option such as Just Enjoy The Story difficulty, you can go easy or hard depends on how much you spent time at farming. It is up to you. Though there were several times that I screamed (lol) for anger, it helped me to get used to manipulating Nintendo Switch controller.

activity


Some old people may know about Touhou Project. It is one of vertically scrolled shooting games such as Strikers 1945. The game was popular on 90s and 00s for its unique worldview, which lead to numerous numbers of derivative works created by users. Some works are still created in these days although it is become less popular than before. But, almost every one is aimed for English or Japanese version not the Korean.

Fortunately, Touhou Spell Bubble has recently started to support Korean version. It was first released at 2020/02/06, and started to support Korean version from 2020/10/15. Despite of many concerns, I was happy to see the game supports Korean. (You might know, Korean market is not actually attractive.) So I was willing to visit the cafe when the game is promoted. We ordered every menu once and pictured them.


I remember that I mentioned the laboratory in previous retrospection. (Maybe because I did my best in lab) The professor of lab suggested me to give a lecture, whose content is about C language. I accepted the suggestion without hesitation, and wrote some slideshows. The lecture is in the first-year curriculum, however, I wanted to deal with real application of C language. Why ? Every student in 2020 may have a doubt on studying C language. Because there are already many programming languages that looks awesome and easy such as Python or Javascript.

So I focused on “Why We Study C Language” and “How C Language Is Used” for resolving the doubts. I prepared the contents like Explaining “Why C Language Can Manipulate Memory” with assembly codes, showing “How C Language Is Used In Real Project” with Linux kernel codes. People rarely say about them. Time has passed, we do not use C language for all purposes. I thought now we should consider to focus what only C can do when we teach C language to students.

work



Already one year passed from join the PUBG. Exactly 1.5 year ? I am now familiar to my work, and even got the sub role additionally. We had an anniversary cake with people who joined PUBG in the same time, too. Work is not easy going, but we try to do boost each other and overcome it. I wish I can go with the people as many as can also even one year passed again. It is sad that now we cannot get together in offline due to the crisis. We had often got together once per 1~2 months, and had a dinner. I already miss that times.

Starting remote work, I found some pros and cons. Remote work seems not the silver bullet one in every situation. Of course, there are common topics on remote work regardless of job. But, some topics are unique ones only existing on game developer job. I can show cons below:

Common Topics

  • chores that did not exist when you commute to office
    (ex: cooking, managing workplace)
  • hardness to know about work mood
    (ex: are they excited ? are they angry ?)
  • irregular work time with personal circumstances
    (ex: family with kids)

Extra Topics (for game developer job)

  • heavy traffic due to massive volume of program
    (it is painful at home network to upload or download programs when compared with office network)
  • laggy remote screen sharing
    (game developer should usually run the client program, while it is okay other developers use only console prompt)
  • poor response time on input or output
    (when the game is kind of real time game…)

Play Station 5 has been released in recent. Sony planned to celebrate the event with partner companies, and collected the picture of members. I sent the picture above, and you can find it on the site https://sie.offbaseproductions.com/ too. It is something monumental and memorable. Sony did good job. I was glad to develop on PS5. :)

thoughts

This year, I leave some regrets that I should have done more things. I should have met more people and read books. But…the lethargy from the crisis, everyone may have felt this. I cannot assert it did not affect me. People around me seems sometimes sad and depressed, too. What was worse, we end up this year as bad situation with high amount of patients. It is hard to believe next year would be better.

Somebody said, “The World Never Be The Same”. At first of this year, I did not agree the words. Because I could not imagine the new world. But…we going to the new world anyway, and it seems we must adapt. Even this year is said the most terrible one, I think we should remember it. To look back to stop this tragedy. We gotta be worry about how to adapt new world and how to live next year, based on new rules.

Gamepad input process in UnrealEngine


Terms

Term Synonym Meaning
Non-Axis Key, Button input type which has only two states: Pressed or Released
Axis Stick input type which has numberless states
Key (sometimes) it is used for indicating any input type. be careful with context for distinction of Non-Axis
Deadzone Threshold range of input values, which blocks values in range. only bigger value cannot be blocked

Gamepad Non-Axis/Axis Mapping in UnrealEngine

1
2
3
4
5
6
7
8
9
10
11
UnrealEngine/Engine/Source/Runtime/Core/Private/GenericPlatform/GenericApplication.cpp

...
const FGamepadKeyNames::Type FGamepadKeyNames::LeftAnalogX("Gamepad_LeftX");
const FGamepadKeyNames::Type FGamepadKeyNames::LeftAnalogY("Gamepad_LeftY");
const FGamepadKeyNames::Type FGamepadKeyNames::RightAnalogX("Gamepad_RightX");
const FGamepadKeyNames::Type FGamepadKeyNames::RightAnalogY("Gamepad_RightY");
const FGamepadKeyNames::Type FGamepadKeyNames::LeftTriggerAnalog("Gamepad_LeftTriggerAxis");
const FGamepadKeyNames::Type FGamepadKeyNames::RightTriggerAnalog("Gamepad_RightTriggerAxis");
...

we can find some pre-defined non-axis/axis keys as FName in GenericApplication.cpp. they are for mapping from various input messages to generic input messages. “various input messages” means, there are many types of gamepad in the world. the button/stick layout differs in Xbox One controller, Playstation 4 controller and so on. the gamepads below are Xbox One, Playstation 4, Stadia and Switch in order.

look at the Xbox One one and Playstation 4 one. they have many differences such as position of stick and exsitance of touch pad. even in comparison for Xbox One one and Switch one, the number of buttons differs. in this situation, it is not easy for every individual developer to support every type of gamepad, so the need of generic mapping for gamepad input arises. let us find out the generic mapping with Xbox One controller examples.

the tables are from the reference #1. you can find more details for each button/stick at the URL. though there are so many items in table, some of them are not counted as user input in common situation. so, we can gotta consider the items below:

Index Item Name Unreal Mapping Input Type
1 Left Stick (Move Horizontally) Gamepad_LeftX Key, Axis
(Move Vertically) Gamepad_LeftY Key, Axis
(Move Left Side More Than Deadzone) Gamepad_LeftStick_Left Key
(Move Up Side More Than Deadzone) Gamepad_LeftStick_Up Key
(Move Right Side More Than Deadzone) Gamepad_LeftStick_Up Key
(Move Down Side More Than Deadzone) Gamepad_LeftStick_Down Key
(Click) Gamepad_LeftThumbstick Key
2 Left Bumper Gamepad_LeftShoulder Key
3 View Button Gamepad_Special_Left Key
6 Menu Button Gamepad_Special_Right Key
7 Right Bumper Gamepad_RightShoulder Key
8 Directional Pad (Left) Gamepad_DPad_Left Key
(Up) Gamepad_DPad_Up Key
(Right) Gamepad_DPad_Right Key
(Down) Gamepad_DPad_Down Key
10 Right Stick (Move Horizontally) Gamepad_RightX Key, Axis
(Move Vertically) Gamepad_RightY Key, Axis
(Move Left Side More Than Deadzone) Gamepad_RightStick_Left Key
(Move Up Side More Than Deadzone) Gamepad_RightStick_Up Key
(Move Right Side More Than Deadzone) Gamepad_RightStick_Up Key
(Move Down Side More Than Deadzone) Gamepad_RightStick_Down Key
(Click) Gamepad_RightThumbstick Key
11 Right Trigger Gamepad_RightTriggerAxis Key, Axis
(Press More Than Deadzone) Gamepad_RightTrigger Key
14 Left Trigger Gamepad_LeftTriggerAxis Key, Axis
(Press More Than Deadzone) Gamepad_LeftTrigger Key
X X Button Gamepad_FaceButton_Left Key
Y Y Button Gamepad_FaceButton_Up Key
A A Button Gamepad_FaceButton_Bottom Key
B B Button Gamepad_FaceButton_Right Key

some of them are handled as not only Key but Axis, too.


  • the Gamepad_LeftY is the one of cases

Gamepad Non-Axis Input Handling Process

focus the function XInputInterface::SendControllerEvents(). there is the logic to filter hardware input state.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
UnrealEngine/Engine/Source/Runtime/ApplicationCore/Private/Windows/XInputInterface.cpp

...

void XInputInterface::SendControllerEvents()
{
...

CurrentStates[X360ToXboxControllerMapping[0]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_A);
CurrentStates[X360ToXboxControllerMapping[1]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_B);
CurrentStates[X360ToXboxControllerMapping[2]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_X);
CurrentStates[X360ToXboxControllerMapping[3]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_Y);
CurrentStates[X360ToXboxControllerMapping[4]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_LEFT_SHOULDER);
CurrentStates[X360ToXboxControllerMapping[5]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_RIGHT_SHOULDER);
CurrentStates[X360ToXboxControllerMapping[6]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_BACK);
CurrentStates[X360ToXboxControllerMapping[7]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_START);
CurrentStates[X360ToXboxControllerMapping[8]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_LEFT_THUMB);
CurrentStates[X360ToXboxControllerMapping[9]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_RIGHT_THUMB);
CurrentStates[X360ToXboxControllerMapping[10]] = !!(XInputState.Gamepad.bLeftTrigger > XINPUT_GAMEPAD_TRIGGER_THRESHOLD);
CurrentStates[X360ToXboxControllerMapping[11]] = !!(XInputState.Gamepad.bRightTrigger > XINPUT_GAMEPAD_TRIGGER_THRESHOLD);
CurrentStates[X360ToXboxControllerMapping[12]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_DPAD_UP);
CurrentStates[X360ToXboxControllerMapping[13]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_DPAD_DOWN);
CurrentStates[X360ToXboxControllerMapping[14]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_DPAD_LEFT);
CurrentStates[X360ToXboxControllerMapping[15]] = !!(XInputState.Gamepad.wButtons & XINPUT_GAMEPAD_DPAD_RIGHT);
CurrentStates[X360ToXboxControllerMapping[16]] = !!(XInputState.Gamepad.sThumbLY > XINPUT_GAMEPAD_LEFT_THUMB_DEADZONE);
CurrentStates[X360ToXboxControllerMapping[17]] = !!(XInputState.Gamepad.sThumbLY < -XINPUT_GAMEPAD_LEFT_THUMB_DEADZONE);
CurrentStates[X360ToXboxControllerMapping[18]] = !!(XInputState.Gamepad.sThumbLX < -XINPUT_GAMEPAD_LEFT_THUMB_DEADZONE);
CurrentStates[X360ToXboxControllerMapping[19]] = !!(XInputState.Gamepad.sThumbLX > XINPUT_GAMEPAD_LEFT_THUMB_DEADZONE);
CurrentStates[X360ToXboxControllerMapping[20]] = !!(XInputState.Gamepad.sThumbRY > XINPUT_GAMEPAD_RIGHT_THUMB_DEADZONE);
CurrentStates[X360ToXboxControllerMapping[21]] = !!(XInputState.Gamepad.sThumbRY < -XINPUT_GAMEPAD_RIGHT_THUMB_DEADZONE);
CurrentStates[X360ToXboxControllerMapping[22]] = !!(XInputState.Gamepad.sThumbRX < -XINPUT_GAMEPAD_RIGHT_THUMB_DEADZONE);
CurrentStates[X360ToXboxControllerMapping[23]] = !!(XInputState.Gamepad.sThumbRX > XINPUT_GAMEPAD_RIGHT_THUMB_DEADZONE);

...
}

...

the next screenshot shows the value of CurrentStates when you press A button in Xbox One gamepad.

we can see the key, deadzone or threshold macro in the code. the macros are defined at XInput.h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
C:/Program Files (x86)/Windows Kits/10/Include/10.0.18362.0/um/XInput.h

...

//
// Constants for gamepad buttons
//
#define XINPUT_GAMEPAD_DPAD_UP 0x0001
#define XINPUT_GAMEPAD_DPAD_DOWN 0x0002
#define XINPUT_GAMEPAD_DPAD_LEFT 0x0004
#define XINPUT_GAMEPAD_DPAD_RIGHT 0x0008
#define XINPUT_GAMEPAD_START 0x0010
#define XINPUT_GAMEPAD_BACK 0x0020
#define XINPUT_GAMEPAD_LEFT_THUMB 0x0040
#define XINPUT_GAMEPAD_RIGHT_THUMB 0x0080
#define XINPUT_GAMEPAD_LEFT_SHOULDER 0x0100
#define XINPUT_GAMEPAD_RIGHT_SHOULDER 0x0200
#define XINPUT_GAMEPAD_A 0x1000
#define XINPUT_GAMEPAD_B 0x2000
#define XINPUT_GAMEPAD_X 0x4000
#define XINPUT_GAMEPAD_Y 0x8000

//
// Gamepad thresholds
//
#define XINPUT_GAMEPAD_LEFT_THUMB_DEADZONE 7849
#define XINPUT_GAMEPAD_RIGHT_THUMB_DEADZONE 8689
#define XINPUT_GAMEPAD_TRIGGER_THRESHOLD 30

...

there is an exception case for input smaller than deadzone. let us take an example with Gamepad_LeftX message. check out screenshots below.

when you input a tiny change on left stick, InputAxis() is called. and the key will be accumulated in EventAccumulator.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
UnrealEngine/Engine/Source/Runtime/Engine/Private/UserInterface/PlayerInput.cpp

...

void UPlayerInput::ProcessInputStack(...)
{
...

Exchange(KeyState->EventCounts[EventIndex], KeyState->EventAccumulator[EventIndex]);

...
}

...

after that, the key in EventAccumulator is moved to EventCounts.

if you keep operating the stick, the key is regarded as down.

in this situation, the key is regarded as released if all keys are flushed.

Gamepad Axis Input Handling Process

the callstack of Axis one is similar to the Non-Axis one. there is a difference of execution at XInputInterface::SendControllerEvents()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
UnrealEngine/Engine/Source/Runtime/ApplicationCore/Private/Windows/XInputInterface.cpp

...

void XInputInterface::SendControllerEvents()
{
...

// Send new analog data if it's different or outside the platform deadzone.
auto OnControllerAnalog = [this, &ControllerState](const FName& GamePadKey, const auto NewAxisValue, const float NewAxisValueNormalized, auto& OldAxisValue, const auto DeadZone) {
if (OldAxisValue != NewAxisValue || FMath::Abs((int32)NewAxisValue) > DeadZone)
{
MessageHandler->OnControllerAnalog(GamePadKey, ControllerState.ControllerId, NewAxisValueNormalized);
}
OldAxisValue = NewAxisValue;
};

const auto& Gamepad = XInputState.Gamepad;

OnControllerAnalog(FGamepadKeyNames::LeftAnalogX, Gamepad.sThumbLX, ShortToNormalizedFloat(Gamepad.sThumbLX), ControllerState.LeftXAnalog, XINPUT_GAMEPAD_LEFT_THUMB_DEADZONE);
OnControllerAnalog(FGamepadKeyNames::LeftAnalogY, Gamepad.sThumbLY, ShortToNormalizedFloat(Gamepad.sThumbLY), ControllerState.LeftYAnalog, XINPUT_GAMEPAD_LEFT_THUMB_DEADZONE);

OnControllerAnalog(FGamepadKeyNames::RightAnalogX, Gamepad.sThumbRX, ShortToNormalizedFloat(Gamepad.sThumbRX), ControllerState.RightXAnalog, XINPUT_GAMEPAD_RIGHT_THUMB_DEADZONE);
OnControllerAnalog(FGamepadKeyNames::RightAnalogY, Gamepad.sThumbRY, ShortToNormalizedFloat(Gamepad.sThumbRY), ControllerState.RightYAnalog, XINPUT_GAMEPAD_RIGHT_THUMB_DEADZONE);

OnControllerAnalog(FGamepadKeyNames::LeftTriggerAnalog, Gamepad.bLeftTrigger, Gamepad.bLeftTrigger / 255.f, ControllerState.LeftTriggerAnalog, XINPUT_GAMEPAD_TRIGGER_THRESHOLD);
OnControllerAnalog(FGamepadKeyNames::RightTriggerAnalog, Gamepad.bRightTrigger, Gamepad.bRightTrigger / 255.f, ControllerState.RightTriggerAnalog, XINPUT_GAMEPAD_TRIGGER_THRESHOLD);

...
}

...

in this case, OnControllerAnalog() is called even a tiny change of input value exists. because the code compares with OldAxisValue != NewAxisValue. the function will not be called only when there is no change on input value.

UnrealEngine unique pointer


overview

as unreal engine handles objects inherits UObject, we can use unreal engine easily. there are several benefits when the object inherits UObject.

  • garbage collection
  • reference update
  • reflection
  • serialization
  • etc.

then you might have one question,
“how we can handle the objects does not inherit UObject ? should we use raw pointer for the objects ?”
well…there are 2 ways for this.

  1. using std::unique_ptr of cpp std library in memory.h
  2. using TUniquePtr of unreal API in UniquePtr.h

you can use std::unique_ptr in unreal project, but unreal engine implements their own smart pointer library. and it is common that using TUniquePtr in unreal project unless you do not need cpp std library.

as purpose and functionality are the same, TUniquePtr is similar to std::unique_ptr. TUniquePtr also provides the unique ownership and other features. let us check out what it is and how it is used.


built-in example

you can find some example in unreal engine code.

1
2
3
4
5
6
UnrealEngine/Engine/Source/Runtime/SandboxFile/Public/IPlatformFileSandboxWrapper.h

class SANDBOXFILE_API FSandboxPlatformFile : public IPlatformFile
{
....
};

class FSandboxPlatformFile is not a class inherits UObject and it is possible to be indicated with TUniquePtr.
( conventionally, prefix U is attached when the class inherits UObject )

1
2
3
4
5
6
7
8
9
10
11
UnrealEngine/Engine/Source/Editor/UnrealEd/Classes/CookOnTheSide/CookOnTheFlyServer.h

UCLASS()
class UNREALED_API UCookOnTheFlyServer : public UObject, public FTickableEditorObject, public FExec
{
....

TUniquePtr<class FSandboxPlatformFile> SandboxFile;

....
};

class UCookOnTheFlyServer is a class inhertis UObject and it contains TUniquePtr with FSandboxPlatformFile as member variable.

1
2
3
4
5
6
7
8
9
10
UnrealEngine/Engine/Source/Editor/UnrealEd/Private/CookOnTheFlyServer.cpp

void UCookOnTheFlyServer::PopulateCookedPackagesFromDisk(const TArray<ITargetPlatform*>& Platforms)
{
....

FString EngineSandboxPath = SandboxFile->ConvertToSandboxPath(*FPaths::EngineDir()) + TEXT("/");

....
}

accessing the object through TUniquePtr is the same on std::unique_ptr. using -> operator, you can access the object as the normal pointer.

1
2
3
4
5
6
7
8
UnrealEngine/Engine/Source/Editor/UnrealEd/Private/CookOnTheFlyServer.cpp

FString UCookOnTheFlyServer::ConvertToFullSandboxPath( const FString &FileName, bool bForWrite ) const
{
check(SandboxFile);

....
}

validation on TUniquePtr is the same on raw pointer. if the value is zero, the TUniquePtr is pointing nullptr. you can use check for ensuring whether the TUniquePtr is valid.

1
2
3
4
5
6
UnrealEngine/Engine/Source/Editor/UnrealEd/Private/Commandlets/AssetRegistryGenerator.cpp

bool FAssetRegistryGenerator::SaveManifests(FSandboxPlatformFile* InSandboxFile, int64 InExtraFlavorChunkSize)
{
....
}

FAssetRegistryGenerator::SaveManifests gets FSandboxPlatformFile* as one of parameters. the type of parameter is not the TUniquePtr, so we should convert TUniquePtr<T> into T*.

1
2
3
4
5
6
7
8
9
10
UnrealEngine/Engine/Source/Editor/UnrealEd/Private/CookOnTheFlyServer.cpp

void UCookOnTheFlyServer::CookByTheBookFinished()
{
....

Generator.SaveManifests(SandboxFile.Get());

....
}

use the TUniquePtr::Get() if you need the raw pointer of TUniquePtr<T>.

1
2
3
4
5
6
7
8
9
10
UnrealEngine/Engine/Source/Editor/UnrealEd/Private/CookOnTheFlyServer.cpp

void UCookOnTheFlyServer::CancelCookByTheBook()
{
....

SandboxFile = nullptr;

....
}

assigning nullptr into TUniquePtr, you can let TUniquePtr release the object and have its value as nullptr.

1
2
3
4
5
6
7
8
9
10
UnrealEngine/Engine/Source/Runtime/SandboxFile/Public/IPlatformFileSandboxWrapper.h

class SANDBOXFILE_API FSandboxPlatformFile : public IPlatformFile
{
....

FSandboxPlatformFile(bool bInEntireEngineWillUseThisSandbox = false);

....
}

constructor of FSandboxPlatformFile takes one parameter as boolean.

1
2
3
4
5
6
7
8
9
10
UnrealEngine/Engine/Source/Editor/UnrealEd/Private/CookOnTheFlyServer.cpp

void UCookOnTheFlyServer::CreateSandboxFile()
{
....

SandboxFile = MakeUnique<FSandboxPlatformFile>(false);

....
}

MakeUnique returns TUniquePtr object and calls the constructor of the template class. in this case, MakeUnique<FSandboxPlatformFile> takes one boolean value.


restriction and caution

suppose you have a class like below

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class UserInfo
{
public:
UserInfo(int32 NewUserID, char* NewUserName)
: UserID(NewUserID), UserName(NewUserName), bInitialized(false)
{
bInitialized = true;
}
virtual ~UserInfo()
{
if (UserName != nullptr)
{
delete[] UserName;
}

bInitialized = false;
}

private:
int32 UserID;
char* UserName;
bool bInitialized;
};

you should only use TUniquePtr at the object, which exists only one thing. unless, you would get an exception for delete on nullptr.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
void SomeUActor::BeginPlay()
{
....

CreateUserInformation();
}

void SomeUActor::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
....

ReleaseUserInformation();
}

void SomeUActor::CreateUserInformation()
{
char* String = new char[100];
strcpy(String, "baemincheon");
UserInformation = MakeUnique<UserInfo>(0, String);

TUniquePtr<UserInfo> AnotherPtr(UserInformation.Get());
}

void SomeUActor::ReleaseUserInformation()
{
UserInformation = nullptr;
}

in this code, CreateUserInformation will be called in BeginPlay and ReleaseUserInformation will be called in EndPlay. UserInformation is a memeber variable with TUniquePtr<UserInfo> type. another TUniquePtr<UserInfo> exists in CreateUserInformation, which gets UserInfo*.

what happens when CreateUserInformation is ended ? AnotherPtr would disappear and its destructor would be called. when the destructor of TUniquePtr is called, it releases memory that TUniquePtr has pointed. as a result, variable UserInformation would be a dangling pointer.

UserInformation has abnormal values.

because already the memory is released, an exception would be thrown when we execute ReleaseUserInformation. it is why you have to use TUniquePtr at the object only existing one thing and care about moving the ownership. moving the ownership with raw pointer is dangerous as we have seen.

1
2
3
4
5
6
7
8
void SomeUActor::CreateUserInformation()
{
char* String = new char[100];
strcpy(String, "baemincheon");
UserInformation = MakeUnique<UserInfo>(0, String);

TUniquePtr<UserInfo> AnotherPtr(MoveTemp(UserInformation));
}

let us move the ownership with MoveTemp API. this code makes AnotherPtr point the memory and UserInformation set the nullptr.

UserInformation has nullptr. so we can avoid the exception.


summary

TUniquePtr is similar to std::unique_ptr and its usage and restriction, too.

  • initialization: you can initialize TUniquePtr with 2 ways
type method
using raw pointer TUniquePtr<T> PointerA(T*);
using other TUniquePtr TUniquePtr<T> PointerA(MoveTemp(PointerB));
  • transfering ownership: for preventing side effect, you should use MoveTemp
1
2
3
TUniquePtr<T> PointerA = new T(...);
TUniquePtr<T> PointerB;
PointerB = MoveTemp(PointerA);
  • release: there are various ways to release the TUniquePtr
1
2
3
4
5
6
7
8
9
10
// way #1 : release implicitly
PointerA = nullptr;

// way #2 : release explicitly
PointerB.Release();

{ // way #3 : using scope
TUniquePtr<T> PointerC = new T(...);
....
} // the destructor of PointerC is called when this block ends

CPP ctor and dtor in UnrealEngine


overview

1
2
3
4
5
6
7
8
9
10
11
UCLASS(config=Game)
class AProjectSandboxCharacter : public ACharacter
{
...

public:
AProjectSandboxCharacter();
virtual ~AProjectSandboxCharacter();

...
}

somebody may have the experience your breakpoint stops in cpp constructor even the object is not placed in the world. you might say,
"Why this happens ? I do not get it why the constructor is called even while opening editor !"

vice versa, destructor is called when editor is closed. in your intuition, the cpp constructor or destructor seems to be called when the actor is created or deleted. but it does not as you saw. then, let us find what do cpp ctor and dtor do in unreal engine. we would get a clue to truth through it.


class default object

unreal engine provides their own macro system, which has several features below:

  • garbage collection
  • reference update
  • reflection
  • serialization
  • etc.

to support these features, unreal engine constructed massive macro magics and hacks. so, cpp in unreal engine acts unlike cpp as we know. this ctor/dtor issue is one of them(different behaviors). from reference #2, we can find the reference of Class Default Object, CDO.

cpp constructor makes the Class Default Object and it is copied whenever you create instance of the UObject. what UObject means in this post, is the object created by NewObject API. from reference #1, we can find the reference of NewObject API.

in summary…

  • if your class(or somewhat) should be utilized with APIs of unreal engine, this must inherit UObject and follow some conventions
  • if you do that, your class gonna have CDO, which is used for cloning object instance
  • in this condition, cpp ctor/dtor only works for CDO

※ for more information of CDO, visit reference #1 and reference #2
※ for more information of Unreal Reflection, visit reference #3


real ctor/dtor for unreal

that is why we could see that the breakpoint stops at cpp ctor/dtor before unreal editor opens. CDO is needed to display the asset editor of the class. try some tests for yourself.


the pictures above says, changes in cpp ctor will be shown in asset editor (if the asset inherits the class). not only the simple float variables, but it affects the various component or material things.

so, let us suppose real ctor/dtor for unreal should do its work when the instance of class is created in “game”, such as spawning bullets when player shoots the gun. there are several APIs for this purpose, but every child of UObject does not have common API.

here is a table for major classes. almost of gameplay framework classes inherit them.

child function
UUserWidget UUserWidget::NativeConstruct
AActor AActor::OnConstruction
UActorComponent UActorComponent::InitializeComponent

summary

  • due to supporting several features, cpp in unreal engine acts unlike cpp as we know.
  • cpp ctor/dtor is for CDO, not for the cloned instances in “game”. there are seperate APIs for them.
    • and it differs upon a class. there is no common unreal ctor/dtor.
  • if you plan to make custom class inherits UObject, you should consider how to make unreal ctor/dtor for it.
    • or just let the class inherits a class already implemented unreal ctor/dtor, such as UUserWidget