changeset 655:53e08e6c5ae6

did: a bit of cleanup with the config, also add some documentation
author Sam <sam@basx.dev>
date Sun, 07 May 2023 00:23:46 +0700
parents 7973a17f76e6
children f06a781e798d
files README.md config.nims semicongine.nimble src/semicongine/buildconfig.nim src/semicongine/settings.nim src/semicongine/telemetry.nim
diffstat 6 files changed, 246 insertions(+), 24 deletions(-) [+]
line wrap: on
line diff
--- a/README.md	Sun May 07 00:22:58 2023 +0700
+++ b/README.md	Sun May 07 00:23:46 2023 +0700
@@ -19,9 +19,8 @@
 Roadmap
 -------
 
-Still tons to do. Making render pipeline and scenegraph somewhat compatible
-seems like it will require quite a bit more of work. Also, audio might require
-quite a bit of work, no experience there (note: audio was super easy to implement)
+Here a bit to see what has been planed and what is done already. Is being
+updated frequently (marking those checkboxes just feels to good to stop working).
 
 Rendering:
 
@@ -31,11 +30,14 @@
 - [x] Per-instance vertex attributes (required to be able to draw scene graph)
 - [x] Fixed framerate
 - [x] Instanced drawing (currently can use instance attributes, but we only support a single instance per draw call)
-- [ ] Textures
+- [x] Textures
+- [ ] Multisampling
+- [ ] Allow different shaders (ie pipelines) for different meshes
+
+Required for 3D rendering:
+
 - [ ] Depth buffering
 - [ ] Mipmaps 
-- [ ] Multisampling 
-- [ ] Allow different shaders (ie pipelines) for different meshes
 
 Asset handling:
 - [ ] Resource concept: load from directory, zip or in-memory-zip, select "mod" as root
@@ -58,7 +60,7 @@
 - [ ] Generic configuration concept (engine defaults, per-user, etc)
 - [ ] Input-mapping configuration
 - [ ] Telemetry
-    - [ ] Add simple event logging service
+    - [x] Add simple event logging service
     - [ ] Add exception reporting
 - [ ] Documentation?
 
@@ -78,3 +80,94 @@
 
 Build-system:
 - [x] move all of Makefile to config.nims
+
+
+Documentation
+=============
+
+Okay, here is first quick-n-dirty documentation, the only purpose to organize my thoughts a bit.
+
+Engine parts
+------------
+
+Currently we have at least the following:
+
+- Rendering: rendering.nim vulkan/*
+- Scene graph: entity.nim
+- Audio: audio.nim audiotypes.nim
+- Input: events.nim
+- Settings: settings.nim
+- Meshes: mesh.nim
+- Math: math/*
+- Telemetry: telemetry.nim (wip)
+- Resources/mods: resources.nim (wip)
+
+Got you: Everything is wip, but (wip) here means work has not started yet.
+
+Configuration
+-------------
+
+Or: How to organize s**t that is not code
+
+Not sure why, but this feels super important to get done right. The engine is
+being designed with a library-mindset, not a framework mindset. And with that,
+ensuring the configuration of the build, runtime and settings in general
+becomes a bit less straight-forward.
+
+So here is the idea: There are three to four different kinds of configurations
+that the engine should be able to handle:
+
+1. Build configuration: Engine version, project name, log level, etc.
+2. Runtime engine/project settings: Video/audio settings, telemetry, log-output, etc.
+3. Mods: Different sets of assets and configuration to allow easy testing of different scenarios
+4. Save data: Saving world state of the game
+
+Okay, let's look at each of those and how I plan to implement them:
+
+**1. Build configuration**
+
+
+**2. Runtime settings**
+
+This is mostly implemented already. I am using the Nim module std/parsecfg.
+There is also the option to watch the filesystem and update values at runtime,
+mostly usefull for development.
+
+The engine scans all files in the settings-root directory and builds a
+settings tree that can be access via a setting-hierarchy like this:
+
+    setting("a.b.c.d.e")
+
+```a.b``` refers to the settings directory ```./a/b/``` (from the settings-root)
+```c``` refers to the file ```c.ini``` inside ```./a/b/```
+```d``` refers to the ini-section inside the file ```./a/b/c.ini```
+```e``` refers to the key inside section ```d``` inside the file ```./a/b/c.ini```
+
+```a.b``` are optional, they just allow larger configuration trees.
+```d``` is optional, if it is not give, ```e``` refers to the top-level section
+of the ini-file.
+
+**3. Mods**
+
+A mod is just a collection of resources for a game. Can maybe switched from
+inside a game. Current mod can be defined via "2. Runtime settings"
+
+I want to support mods from:
+
+a) a directory on the filesystem
+b) a zip-file on the filesystem
+c) a zip-file that is embeded in the executable
+
+The reasoning is simple: a) is helpfull for development, testing of
+new/replaced assets, b) is the default deployment with mod-support and c) is
+deployment without mod-support, demo-versions and similar.
+
+Should not be that difficult but give us everything we ever need in terms of
+resource packaging.
+
+**4. Save data**
+
+Not too much thought here yet. Maybe we can use Nim's std/marshal module. It
+produces JSON from nim objects. Pretty dope, but maybe pretty slow. However, we
+are indie-JSON here, not 10M of GTA Online JSON:
+https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/
--- a/config.nims	Sun May 07 00:22:58 2023 +0700
+++ b/config.nims	Sun May 07 00:23:46 2023 +0700
@@ -21,13 +21,9 @@
 
 proc compilerFlagsDebug() =
   switch("debugger", "native")
-  switch("checks", "on")
-  switch("assertions", "on")
 
 proc compilerFlagsRelease() =
   switch("define", "release")
-  switch("checks", "off")
-  switch("assertions", "off")
   switch("app", "gui")
 
 task single_linux_debug, "build linux debug":
@@ -51,7 +47,6 @@
   setCommand "c"
   mkDir(BUILDBASE & "/" & DEBUG & "/" & WINDOWS)
 
-
 task single_windows_release, "build windows release":
   compilerFlags()
   compilerFlagsRelease()
@@ -59,14 +54,6 @@
   setCommand "c"
   mkDir(BUILDBASE & "/" & RELEASE & "/" & WINDOWS)
 
-task single_crosscompile_windows_debug, "build crosscompile windows debug":
-  switch("define", "mingw")
-  single_windows_debugTask()
-
-task single_crosscompile_windows_release, "build crosscompile windows release":
-  switch("define", "mingw")
-  single_windows_releaseTask()
-
 task build_all_linux_debug, "build all examples with linux/debug":
   for file in listFiles("examples"):
     if file.endsWith(".nim"):
@@ -149,10 +136,8 @@
 task get_vulkan_wrapper, "Download vulkan wrapper":
   exec &"curl https://raw.githubusercontent.com/nimgl/nimgl/master/src/nimgl/vulkan.nim > src/semicongine/vulkan/c_api.nim"
 
-const api_generator_name = "vulkan_api_generator"
-
 task generate_vulkan_api, "Generate Vulkan API":
-  selfExec &"c -d:ssl --run src/vulkan_api/{api_generator_name}.nim"
+  selfExec &"c -d:ssl --run src/vulkan_api/vulkan_api_generator.nim"
   mkDir "src/semicongine/vulkan/"
   cpFile "src/vulkan_api/output/api.nim", "src/semicongine/vulkan/api.nim"
   cpDir "src/vulkan_api/output/platform", "src/semicongine/vulkan/platform"
--- a/semicongine.nimble	Sun May 07 00:22:58 2023 +0700
+++ b/semicongine.nimble	Sun May 07 00:23:46 2023 +0700
@@ -5,7 +5,7 @@
 description   = "Hobby game engine, for games that run on semiconductor engines"
 license       = "MIT"
 srcDir        = "src"
-
+backend       = "c"
 
 # Dependencies
 
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/src/semicongine/buildconfig.nim	Sun May 07 00:23:46 2023 +0700
@@ -0,0 +1,46 @@
+import std/compilesettings
+import std/strutils
+import std/logging
+import std/os
+
+const ENGINENAME = "semicongine"
+
+# checks required build options:
+static:
+  assert querySetting(gc) == "orc", ENGINENAME & " requires --mm=orc"
+  assert compileOption("threads"), ENGINENAME & " requires --threads=on"
+
+  if defined(release):
+    assert compileOption("app", "gui"), ENGINENAME & " requires --app=gui for release builds"
+
+  if defined(linux):
+    assert defined(VK_USE_PLATFORM_XLIB_KHR), ENGINENAME & " requires --d:VK_USE_PLATFORM_XLIB_KHR for linux builds"
+  elif defined(windows):
+    assert defined(VK_USE_PLATFORM_XLIB_KHR), ENGINENAME & " requires --d:VK_USE_PLATFORM_WIN32_KHR for windows builds"
+  else:
+    assert false, "trying to build on unsupported platform"
+
+# build configuration
+# =====================
+
+# compile-time defines, usefull for build-dependent settings
+# can be overriden with compiler flags, e.g. -d:Foo=42 -d:Bar=false
+# pramas: {.intdefine.} {.strdefine.} {.booldefine.}
+
+# root of where settings files will be searched
+# must be relative (to the directory of the binary)
+const DEBUG* = not defined(release)
+const CONFIGROOT* {.strdefine.}: string = "."
+assert not isAbsolute(CONFIGROOT)
+
+const CONFIGEXTENSION* {.strdefine.}: string = "ini"
+
+# by default enable hot-reload of runtime-configuration only in debug builds
+const CONFIGHOTRELOAD* {.booldefine.}: bool = DEBUG
+
+# milliseconds to wait between checks for settings hotreload
+const CONFIGHOTRELOADINTERVAL* {.intdefine.}: int = 1000
+
+# log level
+const LOGLEVEL {.strdefine.}: string = (when DEBUG: "lvlAll" else: "lvlWarn")
+const ENGINE_LOGLEVEL* = parseEnum[Level](LOGLEVEL)
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/src/semicongine/settings.nim	Sun May 07 00:23:46 2023 +0700
@@ -0,0 +1,97 @@
+import std/logging
+import std/parsecfg
+import std/strutils
+import std/sequtils
+import std/parseutils
+import std/strformat
+import std/tables
+import std/os
+
+import ./buildconfig
+
+when CONFIGHOTRELOAD:
+  var configUpdates: Channel[(string, Config)]
+  configUpdates.open()
+
+# runtime configuration
+# =====================
+# namespace is the path from the CONFIGROOT to the according settings file without the file extension
+# a settings file must always have the extension CONFIGEXTENSION
+# a fully qualified settings identifier can be in the form {namespace}.{section}.{key}
+# {key} and {section} may not contain dots
+
+# a "namespace" is the path from the settings root to an *.CONFIGEXTENSION file, without the file extension
+# settings is a namespace <-> settings mapping
+var allsettings: Table[string, Config]
+
+proc configRoot(): string =
+  joinPath(absolutePath(getAppDir()), CONFIGROOT)
+
+proc getFile(namespace: string): string =
+  joinPath(configRoot(), namespace & "." & CONFIGEXTENSION)
+
+iterator walkConfigNamespaces(): string =
+  for file in walkDirRec(dir=configRoot(), relative=true, checkDir=true):
+    if file.endsWith("." & CONFIGEXTENSION):
+      yield file[0 ..< ^(CONFIGEXTENSION.len + 1)]
+
+proc loadAllConfig(): Table[string, Config] =
+  for ns in walkConfigNamespaces():
+    result[ns] = ns.getFile().loadConfig()
+
+proc reloadSettings*() =
+  allsettings = loadAllConfig()
+
+proc configStr(key, section, namespace: string): string =
+  when CONFIGHOTRELOAD:
+    while configUpdates.peek() > 0:
+      let (updatedNamespace, updatedConfig) = configUpdates.recv()
+      allsettings[updatedNamespace] = updatedConfig
+  if not allsettings.hasKey(namespace):
+    raise newException(Exception, &"Namespace {namespace} not found, available namespaces are {allsettings.keys().toSeq}")
+  allsettings[namespace].getSectionValue(section, key)
+
+proc setting*[T: int|float|string](key, section, namespace: string): T =
+  when T is int:
+    let value = configStr(key, section, namespace)
+    if parseInt(value, result) == 0:
+      raise newException(Exception, &"Unable to parse int from settings {namespace}.{section}.{key}: {value}")
+  elif T is float:
+    let value = configStr(key, section, namespace)
+    if parseFloat(value, result) == 0:
+      raise newException(Exception, &"Unable to parse float from settings {namespace}.{section}.{key}: {value}")
+  else:
+    result = configStr(key, section, namespace)
+
+proc setting*[T: int|float|string](identifier: string): T =
+  # identifier can be in the form:
+  # {key}
+  # {section}.{key}
+  # {namespace}.{section}.{key}
+  let parts = identifier.rsplit(".")
+  if parts.len == 1: result = setting[T](parts[0], "")
+  elif parts.len == 2: result = setting[T](parts[1], parts[0])
+  else: result = setting[T](parts[^1], parts[^2], joinPath(parts[0 .. ^3]))
+
+allsettings = loadAllConfig()
+
+when CONFIGHOTRELOAD == true:
+  import std/times
+
+  proc configFileWatchdog() {.thread.} =
+    var configModTimes: Table[string, Time]
+    while true:
+      for namespace in walkConfigNamespaces():
+        if not (namespace in configModTimes):
+          configModTimes[namespace] = Time()
+        let lastMod = namespace.getFile().getLastModificationTime()
+        if lastMod != configModTimes[namespace]:
+          configUpdates.send((namespace, namespace.getFile().loadConfig()))
+      sleep CONFIGHOTRELOADINTERVAL
+  var thethread: Thread[void]
+  createThread(thethread, configFileWatchdog)
+
+if DEBUG:
+  setLogFilter(lvlAll)
+else:
+  setLogFilter(lvlWarn)
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/src/semicongine/telemetry.nim	Sun May 07 00:23:46 2023 +0700
@@ -0,0 +1,1 @@
+# curl -X POST https://semicongine-telemetry.clients1.basx.dev/telemetry/telemetry/event/submit/log/test/1.0/2342343 -v -H "project-api-key: test-key-42"