이제껏, WireFrame 과 Normal 을 봤다. 또한 중요한 부분중에 하나는 Model 이 우리가 봤던 박스처럼 모든 모델이 그렇게 생겨있지 않다. 예를 들어서 구, Cylinder 등의 모형들은 격자 무늬로 이루어져 있으며, 특히나 지형 같은 경우도 Grid Plane 이라고 볼수 있다. 그럼 Grid Plane 은 어떻게 생겨 먹은 친구 인지 한번 봐보자. 아래를 보면, 저런 격자 모양인 Mesh 가 결국엔 Grid Mesh 라고 볼수 있다. 여러개의 박스 (triangle mesh 2개) 가 여러개 모여서 격자 모형의 Plane 을 만들수 있다고 할수 있다.

결국에는 너무 쉽게도 Mesh 의 Vertex 정보와, Normal 정보, 그리고 Vertex 들의 관계 (index) 정보들을 가지고 만들수 있다. 일단 Texture 를 준비해보자. 그리고 현재 Game Engine 에 올려보자. 아래의 Texture 는 물을 표현한 Texture 이다.

그리고 격자를 그리기위해서, 결국에는 Vertex 의 정보가 필요하다. 격자를 그리기위해나 Parameter 로서는 Width, Height, Stack, Slice 라고 보면 될것 같다.

Width & Height 는 격자의 총길이, 그리고 stack 몇개의 Box 를 위쪽으로 쌓을건지와, Slice 는 Width 에서 얼마나 자를건지를 표현한다.

코드는 따로 공유는 하지 않겠다, 하지만 격자는 아래와 같이 그려낼수 있다. 결국에는 평면이기 때문에 임의 Normal 값을 -z (모니터가 나를 바라보는 쪽) 으로 되어있고, 그리고 격자의 Vertex 의 정보는 Slice 와 Stack 으로 Point 를 Translation 해줬으며, 그리고 Index 들은 간단한 offset 으로 구현을 했었다.

재밌으니까, 물결 Texture 니까, 물결을 나타내는 Texture 를 한번 구부려보자. z 축을 x 의 sin graph 로 그려내보자. 그리고 이거에대한 Normal 값도 따로 적용한다고 하면 두개의 Image 를 확인할수 있다. x 에 대한 변화량에 대한 z 값을 그려냈기때문에, 편미분을 통해서 결과값을 도출해낼수 있다.

이제껏 해본걸 종합해보자. Plane 도 만들어보았고, Box Mesh 도 만들어보았다. 근데 Cylinder 는? 사실 Cylinder 는 앞에 Grid 의 평면을 말아 놓은거라고 볼수 있다. 그렇다면, 어떻게 해야할까?가 고민일텐데, Texture 좌표 때문에 Vertex 를 + 1 을 해줘야한다. 그 이유는 Texture 좌표계 때문이다. (0~1) 로 반복되는거로 되어야하기때문에 그렇다.

그리고 앞서서 배웠듯이 Normal Vector 는 inverse transpose 값을 해줘야 Scale 값에 영향이 없는 결과를 가지고 올수 있다.

일단 이거는 살짝의 코드의 방향성을 생각을 해보면 좋을것 같다. 월드좌표계에서 Cylinder 를 만든다고 가정을 했을때, 화면 안으로 들어가는 좌표 z, Right Vector 는 X 축, Up Vector 는 Y 축이라고 생각을하고. 모델링을 만들때는 Y 축을 기준 회전 (즉 x-z 평면에서 만든다고 볼수있다.)

그렇다고 하면, 모든 Vertex 를 얼마나 회전을 하느냐에 따라서, 각도를 const float dTheta = -XM_2PI / float(sliceCount); 결국에는 얼마나 잘라내는지에 따른것에 따라서 더 부드러운 원동모형을 만들수 있을것 같다. 그리고 Y 축의 회전이다 보니, 모든 Vertex 를 Y 축을 통해서 Rotation 값을 누적해가면 된다.

그러면 위의 Radius 와 아래의 Radius 를 x-z 평면으로 시작점으로 해서 돌리면 된다. 결과는 아래와 같다. 원통을 그리려면 Vertex 의 정보를 SimpleMath 에 있는 CreateRotationY 로 충분히 해도 되지만, sin, cos 을 사용해서 원통을 만들어도 똑같은 결과를 나타낸다.

이전 Post 와 마찬가지로, 목적은 우리가 그려야할 Mesh 들의 Wireframe 도 봐야하지만, 각 면에 있는 Face Normal 값을 확인하는것도 Graphics Tool 로서는 중요하다. WireFrame 같은 경우는 DirectX 에서 는 RasterizerState 으로 해주었었다. 하지만 Normal 같은 경우 직접 그려야한다. 그래서 각 Vertex 별 Normal 값을 구하는게 중요하다. 물론 Unreal Engine 같은 경우 아래처럼 World Normal 을 볼수 있게끔 되어있다.

그렇다면 일단 만들어보자. 간단한 Box 의 Vertex 와 Normal 값들을 직접적으로 넣어주는 코드는 생략하겠다. 단 여기서 Point 는 HLSL 에서 어떻게 사용되는지를 알아보는게 더 중요하다.

일단 ConstantBuffer 에 들어가 있는걸로도 충분하니 아래의 코드를 일단 봐보자.

struct VertexShaderInput {
    float3 modelPos: POSITION;
    float3 normalModel: NORMAL;
    float2 texcoord : TEXCOORD0;
}

struct PixelShaderInput {
    float4 posProj : SV_POSITION;
    float3 posWorld: POSITION;
    float4 normalWorld : NORMAL;
    float2 texcoord : TEXCOORD;
    float3 color : COLOR;
}

cbuffer MyVertexConstantBuffer : register(b0)
{
    matrix model;
    matrix invTranspose;
    matrix view;
    matrix projection;
}

PixeShaderInput main(VertexShaderInput input) {
    PixelShaderInput output;
    float4 pos = float4(input.posModel, 1.0f);

    float4 normal = float4(input.normalModel, 0.0f);
    output.normalWorld = mul(normal, invTranspose).xyz;
    output.normalWorld = normalize(output.normalWorld);

    pos = mul(pos, model);

    float t = input.texcoord.x;

    pos.xyz += output.normalWorld * t * scale;

    output.posWorld = pos.xyz;

    pos = mul(pos, view);
    pos = mul(pos, projection);

    output.posProj = pos;
    output.texcoord = input.texcoord;

    output.color = float3(1.0, 1.0, 0.0) * (1.0 - t) + float3(1.0, 0.0, 0.0) * t;

    return output;
}

float4 main(PixelShaderInput input) : SV_TARGET
{
    return float4(input.color, 1.0f);
}

일단 Shader 코드를 살표 보자면, View 로 Transform 하기 이전에, Model 좌표계에서 의 Normal Vector 들을 World 좌표계로 구한다. 그런다음에, 시작점과 끝점을 확실히 하기위해서, texcoord 를 CPU 쪽에서 넘겨줄때 .x 값을 넘겨서 시작과 끝을 알려주는거를 넣어주면 될것 같다. 그리고 t 가 1 면 normal vector 의 원점 (노란색) 그리고 t 가 0 이면, normal vector 의 끝점인 (빨간색) 으로 표시할수 있게한다.

이제 CPU 쪽 작업을 해보자. CPU 에서 보내줄 정보는 GPU 에서의 보내줄 정보와 같다. CPU 쪽에서는 새로운 Normal 값들을 집어넣어야 하기에 정점 정보와 Normal 의 Indices 정보를 넣어서, Buffer 안에다가 채워넣어준다. 그리고 말했던 Normal 의 시작과 끝을 알리는 정보로서 texcoord 에 Attribute 로 집어 넣어준다. 마찬가지로 ConstantBuffer 도 Model, View, Projection 을 원하는 입맛에 맛게끔 집어넣어주면 될것 같다. 그리고 마지막으로 ConstantBuffer 를 Update 만해주면 내가 바라보는 시점에 따라서 Normal 도 같이 움직이는걸 확인할수 있다.

struct MyVertexConstantBuffer
{
    matrix model;
    matrix invTranspose;
    matrix view;
    matrix projection;
}

MyVertexConstantBuffer m_MyVertexConstantData;
ComPtr<ID3D11Buffer> m_vertexConstantBuffer;
ComPtr<ID3D11Buffer> m_vertexBuffer;
ComPtr<ID3D11Buffer> m_indexBuffer;
ComPtr<ID3D11VertexShader> m_normalVertexShader;
ComPtr<ID3D11PixelShader> m_normalPixelShader;
UINT m_indexCount = 0;

//-----------------------------------------------------//
// Init ()
std::vector<Vertex> normalVertices;
std::vector<uint16_t> normalIndices;
for (size_t i = 0; i < vertices.size(); i++){
    auto data = verticies[i];
    data.texcoord.x = 0.0f;
    normalVertices.push_back(data);
    data.texcoord.x = 1.0f;
    normalVertices.push_back(data);

    normalIndices.push_back(uint16_t(2*i));
    normalIndices.push_back(uint16_t(2*i + 1));
}

CreateVertexBuffer(normalVertices, m_vertexBuffer);
m_indexCount = UINT(normalIndices.size());
CreateIndexBuffer(normalIndicies, m_indexBuffer);
CreateConstantBuffer(m_MyVertexConstantData, m_vertexConstantBuffer);

// Then you need to Create Vetex / InputLayout & PixelShader to bind the resources.

//-----------------------------------------------------//
// Update ()
// occluded the (M)odel(V)iew(P)rojection Calculation
UpdateBuffer(m_MyVertexConstantData, m_vertexConstantBuffer);

//-----------------------------------------------------//
// Render()
m_context->VSSetShader(m_normalVertexShader.Get(), 0, 0);
ID3D11Buffer *pptr[1] = {m_vertexConstantBuffer.Get()};

m_context->VSSetConstantBuffers(0, 1, pptr);
m_context->PSSetShader(m_normalPixelShader.Get(), 0, 0);
m_context->IASetVertexBuffers(
    0, 1, >m_vertexBuffer.GetAddressOf(), &stride,
    &offset);
m_context->IASetIndexBuffer(m_indexBuffer.Get(),
                            DXGI_FORMAT_R16_UINT, 0)
m_context->IASetPrimitiveTopology(D3D11_PRIMITIVE_TOPOLOGY_LINELIST);
m_context->DrawIndexed(m_indexCount, 0, 0);

결과는 아래와 같다.

RenderDoc 으로도 돌려봐야하지 않겠냐? 싶어서, Vertex 의 Input 정보들을 확인할수 있다. 그리고 내가 어떠한 Parameter 로 넣어줬는지에 대한 State 들도 왼쪽에 명시 되어있다.

그리고 Vertex Shader 로 부터 Output 이 생성이 되면 아래와 같이 각 Vertex 과 면에 대해서 Normal 값이 나오는걸 확인할수 있다.

그리고 이건 DrawIndexed 의 호출을 동그라미 친것이다. ImGUI 도 쓰기때문에 저 뒤에 두번째는 ImGUI 가 현재 RenderTarget 에 DrawIndexed 를 해주고, 내가 Rendering 하고 싶은 결과는 노란색 두개 이다.

이렇게해서 RenderDoc 을 사용해서 검증을 하고 내가 Pipeline 에잘 넣었는지도 확인할수 있다.

결국에는 우리가 직접 Lighting 을 CPU 에서 계산하는게 아닌 GPU 에서 계산하도록 해야한다, 그럴려면 Shader Programming 으로 할수 있는 방법들을 찾아야한다.

일단 기본적인 어떤 Lighting 을 계산하기 위해서는 물체의 Normal 값이 필요하기 때문에, ModelViewProjection 이라는 Constant Buffer 에다가 InverseTransform 을 넣어줘야한다. 여기서 잠깐! 왜 Model 그대로 Normal 을 구하지 않는지를 물어볼수 있다. 그건 바로 Model 의 Scaling 이 변한다고 했을때, Normal 의 길이(크기)가 달라지기 때문에, Inverse Transpose Model Matrix 를 넣어줘야한다. 처음에는 위치값을 제거하고 (Translation), 그리고 .Invert().Transpose() 를 해주면, World Normal 값을 Shader 에서 구할수 있다.

자 일단 ConstantBuffer 를 Update 해주는 곳은 Render() 에서 매 Frame 별로 Update 를 해준다. 그러기때문에 아래의 방식처럼 CPU 에서 GPU 로 Data 넘길것들을 정의 해준다. 물론 이야기는 따로 하진 않겠지만, Model -> View -> Projection 으로 행렬을 곱하는데, 월드 좌표계에서의 Camera 의 위치를 구하기 위해서, m_pixelConstantBufferData.eyeWorld = Vector3::Transform(Vector3(0.0f), m_vertexConstantBufferData.view.Invert()); 이부분이 들어간거다.

struct Vertex {
    Vector3 position;
    Vector3 normal;
    Vector2 texcoord;
};

struct VertexConstantBuffer {
    Matrix model;
    Matrix invTranspose;
    Matrix view;
    Matrix projection;
};

// Render() -> Constant Buffer Update
m_vertexConstantBufferData.model =
    Matrix::CreateScale(m_modelScaling) *
    Matrix::CreateRotationX(m_modelRotation.x) *
    Matrix::CreateRotationY(m_modelRotation.y) *
    Matrix::CreateRotationZ(m_modelRotation.z) *
    Matrix::CreateTranslation(m_modelTranslation);

m_vertexConstantBufferData.model =
        m_vertexConstantBufferData.model.Transpose();

m_vertexConstantBufferData.invTranspose = m_vertexConstantBufferData.model.;
m_vertexConstantBufferData.invTranspose.Translation(Vecctor3(0.0f)); // get rid of translation
m_vertexConstantBufferData.invTranspose = m_vertexConstantBufferData.invTranspose.Transpose().Invert()

m_vertexConstantBufferData.view = Matrix::CreateRotationY(m_viewRot) * Matrix::CreateTranslation(0.0f, 0.0f, 2.0f);

m_pixelConstantBufferData.eyeWorld = Vector3::Transform(
    Vector3(0.0f), m_vertexConstantBufferData.view.Invert());
m_vertexConstantBufferData.view =
    m_vertexConstantBufferData.view.Transpose();

이렇게 필요한 Data 가 주어졌을떄, Lighting 기반 Bling Phong with Lambert equation 을 사용해서 Shader Programming 을 해야한다. 일단 CPU 쪽의 Data 를 정의 한다. 카메라의 위치와 어떤 Light 인지를 담는, Constant Buffer 를 사용하자. Lights 의 종류는 3 가지 (Directional Light, Point Light, Spotlight) 이렇게 나누어진다.

#define MAX_LIGHTS 3
struct PixelConstantBuffer {
    Vector3 eyeWorld; 
    bool dummmy;
    Material material;
    Light lights[MAX_LIGHTS];
}

struct Light {
    Vector3 strength = Vector3(1.0f);             
    float fallOffStart = 0.0f;                     
    Vector3 direction = Vector3(0.0f, 0.0f, 1.0f); 
    float fallOffEnd = 10.0f;                    
    Vector3 position = Vector3(0.0f, 0.0f, -2.0f); 
    float spotPower = 1.0f;                        
};

struct Material {
    Vector3 ambient = Vector3(0.1f);  
    float shininess = 1.0f;           
    Vector3 diffuse = Vector3(0.5f);  
    float dummy1;                     
    Vector3 specular = Vector3(0.5f); 
    float dummy2;                     
};

class Renderer()
public:
    ComPtr<ID3D11Buffer> m_pixelShaderConstantBuffer;
    ComPtr<ID3D11Texture2D> m_texture;
    ComPtr<ID3D11ShaderResourceView> m_textureResourceView;
    PixelConstantBuffer m_pixelConstantBufferData;
    float m_materialDiffuse = 1.0f;
    float m_materialSpecular = 1.0f;

그리고 Render() 하는 부분에서, ConstantBuffer 값을 Update 를 한다. 빛의 종류에 따른 Update. Directional Light = 0, Point Light = 1, Spot Light = 2. 이렇게 되어있으며, 내가 필요한 Light 만 사용할수 있게끔 다른 Light 들을 꺼주는것이다. (.strength *= 0.0f)

m_pixelConstantBufferData.material.diffuse = Vector3(m_materialDiffuse);
m_pixelConstantBufferData.material.specular = Vector3(m_materialSpecular);
for (int i = 0; i < MAX_LIGHTS; i++) {
    if (i != m_lightType) {
        m_pixelConstantBufferData.lights[i].strength *= 0.0f;
    } else {
        m_pixelConstantBufferData.lights[i] = m_lightFromGUI;
    }
}

CPU 쪽에서 Data 를 만들었다면 이제 GPU 에다가도 똑같이 Resource Binding 을위해서 정의를 한다.

// Common.hlsli
#define MAX_LIGHTS 3 
#define NUM_DIR_LIGHTS 1
#define NUM_POINT_LIGHTS 1
#define NUM_SPOT_LIGHTS 1

struct Material
{
    float3 ambient;
    float shininess;
    float3 diffuse;
    float dummy1; 
    float3 specular;
    float dummy2;
};

struct Light
{
    float3 strength;
    float fallOffStart;
    float3 direction;
    float fallOffEnd;
    float3 position;
    float spotPower;
};

float CalcAttenuation(float d, float falloffStart, float falloffEnd)
{
    // Linear falloff
    return saturate((falloffEnd - d) / (falloffEnd - falloffStart));
}

float3 BlinnPhong(float3 lightStrength, float3 lightVec, float3 normal,
                   float3 toEye, Material mat)
{
    float3 halfway = normalize(toEye + lightVec);
    float hdotn = dot(halfway, normal);
    float3 specular = mat.specular * pow(max(hdotn, 0.0f), mat.shininess);
    return mat.ambient + (mat.diffuse + specular) * lightStrength;
}

float3 ComputePointLight(Light L, Material mat, float3 pos, float3 normal, float3 toEye) {
    float3 lightVec = L.position - pos;
    float d = length(lightVec);

    if (d > L.fallOffEnd) {
        return float3(0.0, 0.0, 0.0);
    } else {
        lightVec /= d;
        float ndotl = max(dot(ligthVec, normal), 0.0f); // Light STrength related
        float3 lightStrength = L.strength * ndotl;
        float attenutationFactor = CalcAttenuation(d, L.fallOffStart, L.fallOffEnd);
        lightStrength *= attenutationFactor;
        return BlingPhong(lightStrength, lightVec, normal, toEye, mat);
    }
}

위를 코드를 사용해서, 각각의 Directional Light / Point Light / Spot Light 를 구현할수 있을것이다. 추가적으로 Spot Light 같은경우에는, Pow 를 사용해서, Alpha 값을 제곱을 해주는 작업도 필요하다.

Resource

• ScratchPixel

가끔 우리는 어떤 Mesh를 사용하고 있는지, 그리고 그 Mesh가 얼마나 많은 삼각형으로 이루어져 있는지를 파악해야 할 필요가 있다.. 이는 성능 최적화나 실시간 렌더링 가능 여부를 평가할 때 중요하고 특히, 고해상도 Terrain이나 복잡한 조각상 같은 모델이 있는 경우 (물론 다른 stage 에서..), 런타임에서 모델을 사용할 수 있는지 판단하는 데 중요한 기준이 됩니다.

예를 들어, Unreal Engine의 Nanite System은 매우 많은 Vertex를 활용하여 고도로 디테일한 모델을 실시간으로 렌더링할 수 있다.. 이는 LOD(레벨 오브 디테일)를 효율적으로 관리하여 성능 저하 없이도 높은 품질의 그래픽을 제공할 수 있게 해줍니다. 그렇다면, 실제로 삼각형 개수를 확인하려면 어떻게 해야 할까요? 정답은 Rendering Pipeline의 상태를 확인하는 것입니다. 특히, Rasterizer 단계에서 어떻게 삼각형을 그릴 것인지 결정하는 RasterizerState를 통해 설정할 수 있다. 그리고 나중에 Rasterizer State 들을 만들어주면 된다.

// Render 
ComPtr<ID3D11RasterizerState> _rasterizerState;
m_context->RSSetStet(_rasterizerState);

// INIT()
D3D11_RASTERIZER_DESC rastDesc;
ZeroMemory(&rastDesc, sizeof(D3D11_RASTERIZER_DESC)); 
m_OnWireFrame ? rastDesc.FillMode = D3D11_FILL_MODE::D3D11_FILL_WIREFRAME : rastDesc.FillMode = D3D11_FILL_MODE::D3D11_FILL_SOLID;
rastDesc.CullMode = D3D11_CULL_MODE::D3D11_CULL_NONE;
rastDesc.FrontCounterClockwise = false;
rastDesc.DepthClipEnable = true; 
m_device->CreateRasterizerState(&rastDesc,
                                 m_solidRasterizerSate.GetAddressOf());
 
m_device->CreateRasterizerState(&rastDesc,
                                 m_wireRasterizerSate.GetAddressOf());

물론 더나아가서 고품질의 그림을 그릴수도 있지만, 일단 WireFrame 을 그리는거까지 해보는것이 중요한것 같다. 결과는 아래와 같다.

내가 Build System 을 고려하다가 정리를 해본다.

지원 플랫폼 및 유연성

• Premake: 주로 Visual Studio, Xcode, GNU Make를 타겟으로 사용하며, 게임 엔진이나 그래픽 라이브러리 개발에 많이 사용됩니다.
• CMake: 거의 모든 플랫폼과 빌드 시스템을 지원합니다. 특히 Ninja와 함께 사용할 때 빌드 속도가 매우 빠릅니다.

  생태계와 커뮤니티 지원

• Premake: 게임 엔진 및 일부 그래픽 라이브러리에서 주로 사용됩니다. 대표적으로 LunaEngine이나 Unreal Engine에서 사용합니다.
• CMake: 오픈소스 프로젝트에서 표준으로 사용됩니다. 예를 들어 LLVM, Qt, OpenCV 등이 CMake를 사용

I’m currently working on my own game engine, called Luna Game Engine. Sometimes, the workload can feel overwhelming, especially when balancing personal projects and portfolio building. I find that taking a break and reflecting helps me maintain focus and productivity.

While working on Luna, I often draw inspiration from existing game engines like Unreal Engine, Unity, and FrostBite. Each of these engines has a distinct philosophy and approach to game development. For example, Unity is particularly focused on multi-platform support, which makes it versatile for mobile, console, and desktop applications. There’s a great video on YouTube that outlines Unity’s roadmap and how they plan to evolve the engine.

One remarkable feature of Unreal Engine is its support for large-scale triangle meshes through Nanite. However, I am a bit concerned about the runtime performance when loading these heavy meshes. While the visuals are undeniably impressive, maintaining performance is always a critical consideration.

Unity’s collaboration with Zyva is also noteworthy, especially when it comes to realistic facial animations. The demo shows incredibly vivid muscle movements that mimic human expressions. This kind of technology could significantly impact the VR industry and even healthcare applications.

Personally, I have a soft spot for FrostBite, as I have enjoyed the Battlefield series for years. I also find it fascinating how the engine has evolved, especially for FIFA. In older versions, scoring from long-distance shots was quite easy, but recent iterations have introduced more nuanced physics and player attributes, making gameplay more realistic. Small details, like players sweating, might seem minor, but they reflect just how far game engines have come in terms of visual fidelity and realism.

As I continue developing Luna Game Engine, I aim to incorporate some of the strengths I admire in these established engines while maintaining my own vision. Taking time to study how industry leaders approach engine design helps me make more informed decisions about Luna’s features and structure.

The Vulkan is the graphical API made by Khronos providing the better abstraction on newer graphic cards. This API outperform? the Direct3D and OpenGL by explaining what it perform. The idea of the Vulkan is similar to the Direct3D12 and Metal, but Vulkan is cross-platform, and it can be used and developed in Linux or Window Environment.

However, the drawback of this could be is there will be many detailed procedure while using the Vulkan API, such as creating the buffer frame, managing the memory for buffer and texture image objects. That means we would have to set up properly on application. I realized that the Vulkan is not for everyone, it is only for people who’s passionate on Computer Graphics Area. The current trends for computer graphics in Game Dev, they are switching the DirectX or OpenGL to Vulkan : Lists of Game made by Vulkan. One of the easier approach could be is to learn and play the computer graphics inside of Unreal Engine and Unity.

By designing from scratch for the latest graphics architecture, Vulkan can benefit from improved performance by eliminating bottlenecks with multi-threading support from CPUs and providing programmers with more control for GPUs. Reducing driver overhead by allowing programmers to clearly specify intentions using more detailed APIs, and allow multiple threads to create and submit commands in parallel and Reducing shader compilation inconsistencies by switching to a standardized byte code format with a single compiler. Finally, it integrates graphics and computing into a single API to recognize the general-purpose processing capabilities of the latest graphics cards.

There are three preconditions to follow

• Vulkan (NVIDA, AMD, Intel) compatible graphic cards and drivers
• Expereince in C++ ( RAII, initializer_list, Modern C++11)
• Compiler above C++17 (Visual Studio 2017, GCC 7+, Clang 5+)
• 3D computer Graphics Experience

How to install Vulkan on Visual Studio Code

Have to figure this out later on.

• How to build c++ code in visual studio code
• How to set the environment in visual studio code

Vulkan Componenet

Resources:

• Vulkan Tutorial
• Vulkan Tutorial Youtube-1
• Vulkan Tutorial Youtube-2
• Vulkan Tutorial Youtube-3
• Vulkan Introduction

This is just a graphics resource to learn DirectX

• DirectX Tutorial.com
• Braynzar Soft
• Rastertek.com
• ChillTomatoNoodle Youtube
• Cherno
• Learning Vulkan Github Repo
• MoltenVK Github Repo
• ImGUI - DirectX12
• ImGUI - Vulkan
• Mega3D

Limitation on Vision Transformer (ViT)

While Vision Transformers (ViT) have shown strong performance in image classification. There are several limitations that arise when applying them to more and various computer vision tasks.

1. Fixed Patch Size & Lack of Local Content

ViT splits the image into fixed-size patches (e.g., 16x16), regardless of the underlying content

As a results:

• The model may attent to meaningless or background regions (e.g white pixels) with no semantic values
• If important visual elements are split across patch boundaries, local similarity between pixels is lost

Maybe the example I can think of is, a small object like a bird might get split between patches, causing the network to miss its continuity entirely.

2. Poor Handling of Scale Variations

ViT lacks mechanisms to handle scale variablity in visual tokens. In real-world images:

• A bird could occupy 200x200 or just 20x20, but both should still be recognized as “a bird”
• Without scale-awareness, the model struggles to generalize across different object sizes.

3. Quadratic Complexity

ViT’s self-attention has quadratic complexity with respect to the number of patches (i.e., ( O(n^2) )).

This becomes computationally expensive for high-resolution images commonly found in practice (e.g., 1920×1080).

4. Limited to Classification Tasks

ViT was initially designed for image classification, but many vision applications require more:

• Object Detection: Unlike language tokens, visual elements vary widely in size and spatial distribution.
• Semantic Segmentation: Image resolution is much higher than text sequences, requiring dense predictions across pixels.

Solution & Architecture: Hierarchical Structure & Shifted Window Attention

• A pyramid-like hierarchical structure, inspired by CNNs and U-Net
• Shifted window-based local self-attention, which allows the model to:
  • Capture local context efficiently
  • Reduce computation to linear complexity with respect to image size
  • Scale well to high-resolution and dense prediction tasks

These architectural choices allow Transformers to be applied beyond classification—including detection and segmentation

A General Transformer Backbone for Vision

The diagram above illustrates the general structure of a Transformer-based vision backbone. The Swin Transformer begins by splitting the input image into small non-overlapping patches (gray outlines) and gradually builds up hierarchical representations by merging neighboring patches at deeper layers—similar to CNNs.

Local Self-Attention with Shifted Windows

To reduce computational complexity, Swin Transformer performs self-attention within non-overlapping windows (red outlines) rather than across the entire image.

• These windows are fixed in size (e.g., 4×4 patches).
• Attention is calculated only within each window, not globally.
• As a result, computational complexity becomes linear with respect to image size, instead of quadratic.

This window-based local attention allows the model to efficiently process images of various sizes, since the number of windows scales with the image size.

Example: Window and Patch Hierarchy

• Suppose the input image is divided into 16×16 windows, each window containing 4×4 patches.
• Within each 4×4 window:
  • Self-attention is computed locally, without looking outside the window.
• In deeper layers:
  • Neighboring patches are merged (e.g., combining 2×2 patches).
  • This reduces the total number of patches by a factor of 4, while increasing the spatial resolution of each patch (i.e., width × 2 and height × 2).

This hierarchical merging process enables the model to learn increasingly abstract and global features, while maintaining efficiency and scalability.

Multi-head Self-Attention in Swin Transformer

In Swin Transformer, attention is applied locally within windows. However, instead of applying the same partitioning across all layers, it introduces a clever mechanism:
Shifted Windows, which allows the model to connect neighboring windows and capture richer context. Each patch in a window shares the same key set, which not only simplifies computation but also improves memory access efficiency—crucial for hardware acceleration as explain furthermore.

Notably, all windows in Swin are non-overlapping.

W-MSA (Window-based Multi-head Self-Attention)

In Layer 1 of the Swin Transformer block (left side of the original figure above), the image is divided into non-overlapping windows. Self-attention is applied within each window only, making it a local attention mechanism. This is referred to as W-MSA.

SW-MSA (Shifted Window-based Multi-head Self-Attention)

In Layer 2 (right side of the original figure), the windows are shifted relative to the previous layer. This allows self-attention to be computed across window boundaries, connecting adjacent regions. While traditional sliding windows can also cover neighboring areas, shifted windows maintain efficiency while introducing inter-window connections, enabling the model to capture global interactions gradually.

Traditional Multi-head Self-Attention (MSA) computes attention globally across all tokens in an image. While this enables long-range dependencies, it comes with a significant computational cost.

Before, the complexity of global self-attenion is:

MSA: Ω(MSA) = 4 * h * w * C^2 + 2 * (h * w)^2 * C

Where:

• h, w = height and width of the feature map
• C = number of channels

But there is a problem with this, if the input resolution increases, the quadratic term grows dramatically - making it inefficient for high-resolution images.

To address this, Window-based Multi-head Self-Attention (W-MSA) restricts attention to local windows of size M × M. Its complexity is:

W-MSA: Ω(W-MSA) = 4 * h * w * C^2 + 2 * M^2 * h * w * C

Here, M is a fixed window size (e.g., 7), making this approach linear in terms of image size. This dramatically reduces computational cost while retaining performance in local contexts.

Also, the each swin transformer block consists of:

• A W-MSA or SW-MSA module
• A two-layer MLP with GELU activation
• Layer Normalization before each sub-layer
• A residual connection (skip connection) applied after each block

Model Architecture

The figure above illustrates the overall architecture of the Swin Transformer. Here’s how the input image is processed step-by-step:

1. Patch Partitioning
   The input image is first split into non-overlapping patches of size 4×4, resulting in patch tokens of shape 4×4×3. Each patch is then flattened and passed through a Linear Projection to form an embedding vector.

2. Linear Embedding
   These patch vectors are embedded into a fixed-dimensional space using a learnable linear layer. This prepares them for the Transformer blocks that follow.

3. Transformer Blocks (Stage 1)
   The embedded patches are fed into Transformer blocks, where self-attention is computed both within patches (local) and between patches (global). This stage captures fine-grained, low-level features.

4. Patch Merging (Stage 2)
   After Stage 1, a Patch Merging Layer is applied. This merges each group of 2×2 neighboring patches, concatenating their features into a single vector (dimension becomes 4C). A linear layer then reduces this to 2C, effectively reducing spatial resolution (by a factor of 2) and increasing the channel capacity.

5. Hierarchical Feature Learning (Stages 3 & 4)
   This patch merging process and Transformer block application are repeated multiple times, forming deeper stages. As a result, the feature maps get smaller in spatial dimensions but richer in representation—similar to how UNet or image pyramids work in traditional vision architectures.

6. Final MLP Head
   At the end of the final stage, a Multi-Layer Perceptron (MLP) head is applied to perform the final prediction task, such as classification or detection.

To explain furthermore on second layer, let’s look at the image below.

Shifted Windows in Swin Transformer

Let’s slowly wrap this, Swin Transformer, introduces a novel mechanism—Shifted Window Multi-head Self-Attention (SW-MSA)—to efficiently model long-range dependencies without incurring the high cost of global self-attention.

Why Shift Windows? The baseline attention mechanism, Window-based MSA (W-MSA), computes self-attention within non-overlapping windows. This is efficient, but it lacks cross-window communication.

To address this, shifted windows are applied in alternating Transformer blocks:

• Each window is shifted by a fixed size (e.g., half the window dimension).
• This causes adjacent windows to partially overlap.
• As a result, tokens from different regions can now interact, enhancing the model’s representational power.

Cyclic Shift & Attention Masking

Shifting windows introduces new windows and disrupts alignment, potentially increasing computational complexity. Swin Transformer handles this elegantly using:

• Cyclic Shift:
  • Instead of moving data in memory, patches are cyclically shifted (like a ring buffer), preserving the GPU’s memory alignment and restoring the original number of windows. (ex: a left-side window is shifted down and right, aligning back to the initial layout.)
• Attention Masking:
  • After shifting, some patches may come from semantically unrelated areas.
  • Example: patches A, B, and C are visually close after the shift, but not contextually related.
  • A binary attention mask ensures that attention is only computed within valid, local regions.
  • After self-attention, the cyclic shift is reversed, restoring spatial structure.

Relative Positional Encoding

Unlike traditional Vision Transformers (ViT), which use absolute positional embeddings, Swin Transformer applies:

• No positional encoding at the input.
• Instead, relative position bias is learned and added during attention computation.
  • This bias represents the relative distance between tokens inside each window.
  • It enables the model to maintain spatial structure without global positional references.

• The details are shwon below

  $Attention(Q, K, V) = SoftMax(\frac{QK^T}{\sqrt{d}} + B)Vs$

In conclusion, as shown in the figure above, the Relative Position Bias is generated using $\hat{B}$. Ultimately, for the matrices $Q$, $K$, and $V$, where $M^2$ represents the number of Window Patches, and the relative positions along each axis range from $[-M+1, M-1]$, it can be seen that a smaller-sized Bias matrix $\hat{B}$ is parameterized, and the values of $B$ are derived from $\hat{B}$.

Resource

• Small Object Detection for Birds with Swin Transformer
• Swin Transformer: A Scalable Vision Transformer That just make sense
• Swin Transformer V1 Review
• Swin Transformer Paper Review
• Swin Transformer Paper Review
• Youtube - Swin Transformer Animated and Explained
• Youtube - Hierarchical Vision Transformer using Shifted Window