Scripting in D2K has come a long way. The interface is coming along nicely, largely because Lua makes it so easy to write bindings. For example, this is the function used to match key events to key presses:

static int XI_InputEventIsKeyPress(lua_State *L) {
  event_t *ev = (event_t *)luaL_checkudata(L, 1, "InputEvent");
  int key = luaL_checkint(L, 2);

  if (ev->type == ev_key && ev->pressed && ev->key == key)
    lua_pushboolean(L, true);
  else
    lua_pushboolean(L, false);

  return 1;
}

XI_InputEventIsKeyPress is a method inside the InputEvent class, which is registered like this:

X_RegisterType("InputEvent", 21, 
  "new",                  XI_InputEventNew,
  "reset",                XI_InputEventReset,
  "__tostring",           XI_InputEventToString,
  "is_key",               XI_InputEventIsKey,
  "is_mouse",             XI_InputEventIsMouse,
  "is_joystick",          XI_InputEventIsJoystick,
  "is_mouse_movement",    XI_InputEventIsMouseMovement,
  "is_joystick_movement", XI_InputEventIsJoystickMovement,
  "is_joystick_axis",     XI_InputEventIsJoystickAxis,
  "is_joystick_ball",     XI_InputEventIsJoystickBall,
  "is_joystick_hat",      XI_InputEventIsJoystickHat,
  "is_movement",          XI_InputEventIsMovement,
  "is_press",             XI_InputEventIsPress,
  "is_release",           XI_InputEventIsRelease,
  "get_key",              XI_InputEventGetKey,
  "get_key_name",         XI_InputEventGetKeyName,
  "get_value",            XI_InputEventGetValue,
  "get_xmove",            XI_InputEventGetXMove,
  "get_ymove",            XI_InputEventGetYMove,
  "get_char",             XI_InputEventGetChar,
  "is_key_press",         XI_InputEventIsKeyPress
);

The implementation of X_RegisterObjects is a little long, but it’s not terribly complex.

All this scaffolding enables a simple, but powerful scripting interface. For example, here’s the console’s event handler:

function Console:handle_event(event)
  if event:is_key_press(d2k.Key.BACKQUOTE) then
    self:toggle_scroll()
  end

  if self.scroll_rate < 0 or self.height == 0 then
    return
  end

  if d2k.KeyStates.shift_is_down() then
    if event:is_key_press(d2k.Key.UP) then
      self.output:scroll_up(Console.HORIZONTAL_SCROLL_AMOUNT)
      return true
    elseif event:is_key_press(d2k.Key.PAGE_UP) then
      self.output:scroll_up(Console.HORIZONTAL_SCROLL_AMOUNT * 10)
      return true
    elseif event:is_key_press(d2k.Key.DOWN) then
      self.output:scroll_down(Console.HORIZONTAL_SCROLL_AMOUNT)
      return true
    elseif event:is_key_press(d2k.Key.PAGE_DOWN) then
      self.output:scroll_down(Console.HORIZONTAL_SCROLL_AMOUNT * 10)
      return true
    end
  elseif event:is_key_press(d2k.Key.UP) then
    self.input:show_previous_command()
  elseif event:is_key_press(d2k.Key.DOWN) then
    self.input:show_next_command()
  elseif event:is_key_press(d2k.Key.LEFT) then
    self.input:move_cursor_left()
  elseif event:is_key_press(d2k.Key.RIGHT) then
    self.input:move_cursor_right()
  elseif event:is_key() and event:is_press() then
    self.input:insert_text(event:get_char())
  end

  return false
end

Never mind the temporary placeholder constants (Console.HORIZONTAL_SCROLL_AMOUNT, etc.) ;). The point is that it was very easy to build an entirely new object-oriented event handling interface.

I did have to refactor nearly all the SDL event stuff in C, but it was in need anyway. As a result, after events are pulled from SDL in C, event dispatch and handling happens in Lua now. This means I can slowly start to move things like the menu and screen drawing (NOT the renderer) into Lua, and delete reams of junk code in the process (man, I dream of the day the C menu code dies).

Now that event handling happens in Lua, interfaces can be implemented using only scripting. The console is my test case; as I develop it, I’m learning to use Lua GObject Introspection and fixing the scripting interface I’ve built so far. It’s not a perfect test (there’s not a ton of interaction with D2K-proper), but it’s pretty good for a first test.

Speaking of Lua GObject Introspection, I’m making progress building D2K’s dependencies on Windows. Mingw-builds provides a native, MinGW-w64-built Python which can be used for GObject Introspection, so I’ve been going through the process of writing a bunch of scripts to automatically build everything D2K needs on Windows. This has been a huge roadblock. My secret dream is that peopel can use my work as a kind of cross-platform shim for C/C++ projects, but given the relative unpopularity of similar projects (win-builds, MSYS2, MXE), I’m not optimistic, haha.

Having developed a fair amount of code in Lua, I feel qualified – nay, compelled – to give an opinion!

Lua as an API is fantastic. I wish the documentation were slightly better, but honestly it’s pretty good. The Lua site itself is hideous; I remember better pages from 1998. It sounds like I’m kidding, but I am not. I wish there were better errors in Lua; while they technically make sense, you only realize why they make sense once you guess correctly as to what the problem is. Lua the language is overall a fairly positive experience. The “different just to be different” stuff like ~= meaning ‘not equal to’, elseif instead of literally anything else (else if, elif), and arrays starting at 1 are all extremely bothersome. Especially the array index thing, my God what an abomination. For loop syntax is a little free-form to me; I really don’t understand why

for i=0,10,2 do
  print(string.format('i is %d', i))
end

is better than:

``c for (int i = 0; i <= 10, i += 2) { printf("i is %d\n", i); }

I do understand why it’s worse though: it’s less explicit.

local is a big mistake too. Lua argues that “Local by default is wrong. Maybe global by default is also wrong, [but] the solution is not local by default”, but come on:

• I am typing local everywhere
• In Python, I type global only when I have to, and it serves as documentation
• If you forget to type local, you can destroy your program in a very hard-to-discover way
• If you forget to type global, the damage is limited to that function and whatever depends on it. Which, admittedly, is still pretty bad, but what is better, more damage that’s hard to reason about, or less damage that’s easier to reason about?
• If you meant “declarations should look like declarations”, var would have sufficed, and wouldn’t get conflated with the scoping issue.
• If you meant “scope should be explicit”, how much more explicit than “I defined this variable in this scope, therefore that’s where it lives” can you get? Do you want to define variables for other scopes? MADNESS

The other annoying thing in Lua is that while syntactic sugar abounds for things that really aren’t that big a deal (using : instead of . to pass the implicit self to object method calls, omitting parentheses when only passing a single string/table argument, etc.), there is no syntactic sugar for implementing a class or building a module. You have to wade into this morass of metatables and the implicit workings of require in order to access basic functionality. You can’t argue that Lua isn’t object-oriented because it has : and has a whole section on “Object-Oriented Programming” in the book (complete with helpful APIs for defining userdata types). By the way, “userdata” is really poorly named. Regular userdata is a blob managed by Lua, light userdata is a blob – tracked by a pointer – managed in native code. Boom, that’s it. Anyway, there ought to be an easier way to define classes and modules without understanding the deep workings of require and Lua’s method dispatch.

Lua is also a little too flexible. Leaving off parentheses is just a huge problem for readability, and in a language that makes you type so much (“function”, “then”, “end”, “do”, “repeat”, “until”) it doesn’t make a lot of sense. I guess maybe they were worried about weird symbols confusing users, but how much does dropping the parentheses really buy you when 99% of the time what follows is a “{” ? Plus, while I pretty much chose Lua over Python because it isn’t whitespace-sensitive, there are times I really wish it were. If I read another piece of code with all the if blocks on the same line, I’m gonna burn Github to the ground. Whew!

In fact, those three annoyances pretty much sum up the Lua experience on the script side. Lua is everything you want in a scripting language: fast, simple, flexible and powerful. But then you find a bunch of code written in a totally unreadable style and you get pissed, or you find yourself mucking about with Lua’s internals because there’s no syntactic sugar for this very simple thing you want to do, or your code doesn’t work because Lua does something different for absolutely no reason.

All that said, if you can live with 1-based indexing, Lua is an excellent language. None of the other annoyances are problematic enough to justify not choosing it.

I had originally planned to use Lua to re-implement D2K’s console, but using GObject Introspection via Lua GObject Introspection was just too hard, because GObject Introspection depends on Python, which requires dozens of patches to cross-compile using MinGW-w64. You can find these patches scattered everywhere, from random GitHub or Bitbucket accounts to the MinGW-Builds tarballs, and what’s clear from the number of patches is that none of it is guaranteed to work at all, let alone work correctly.

So now my plan is to essentially build my own bindings. I can only imagine what’s really involved with this, so it may be that I abandon this as well.

Designing as I write this, I envision something like:

=======================================================================
|                                                                     |
|                       xf.Console (in Lua)                           |
|                           ||                                        |
|                           \/                                        |
|                     Widget Library (in Lua)                         |
|                           ||                                        |
|                           \/                                        |
|                       PangoCairo                                    |
|                         /    \                                      |
|                     Pango    Cairo                                  |
|                                                                     |
=======================================================================

This is, unfortunately, lots of work. I’m currently planning on not providing full bindings to Pango, Cairo or PangoCairo, although I don’t know how hard it would be to do so. If it ends up that it’s as simple as generating (or C-N-P’ing) the rest of the code, then I’ll go the whole way. I certainly won’t hold up D2K development for 100% coverage here.

I would feel remiss if I left this post without noting my dismay at Python’s lack of MinGW support, which is starting to snowball into a general dislike of Python. Its abject slowness, inability to use more than 1 CPU core, wild memory consumption, total mishandling of Unicode, lack of support for MinGW, and insane Python3 redesign (for essentially no good reason, because the Unicode redesign is broken) rule it out for almost anything I will ever do. I’d rather build a GUI in C++ with wxWidgets or QT, or C and GTK than I would with Python and respective bindings. I’d sooner build a website in Go or Java than I would with Python and its frameworks. Its CPU and RAM usage rule it out for systems programming, where C/C++/Go/Java shine.

These days, all I use Python for are when I need better scripting than what Bash will provide, and to be honest, I’m thinking about improving my Bash knowledge so I won’t have to do that either. It’s crazy because I have a lot of deep Python knowledge, but it’s just not useful anymore.

Anyway, I think this delays scripting and the console for a while. I was hoping for a quick translation between the existing C code and the (to be rewritten more performantly) new Lua implementation, but it looks like there’s some plumbing I have to add first. That’s a shame because I was excited about this, but now it just looks like 90% grunt work.

Rev. 42f4ae5 lessens the amount of data sent for certain types of actors (plasma, spider demon plasma, bullet puffs and blood spots) when they spawn. This was necessary because these actors tend to spawn very rapidly, causing lots of new information to be sent over the wire (delta compression doesn’t help here because the actors are new), which would lag (throttle) clients on bad connections. A little more could be done here, and in fact, the solution could be seen as a hack. I think Valve, for example, has fine-tuned controls about what entity fields are synchronized between client and server, which can be helpful for mods. In D2K, if you add more rapidly-spawning actors, there’s not a great way to tell the engine to only send certain data about them, let alone specify what that data should be, and how to fill in the blanks clientside.

D2K struggled with this more than other C/S ports because of its “send everything” philosophy. Delta compression works because the world usually changes very little, but an actor is 312 bytes, and delta compression can’t leave any of that data out when an actor first spawns. 2-3 new actors over a couple of TICs isn’t a big deal, but the SSG spawns something like 20 bullet puffs; that’s over 6K!

To fix this, we pull the size down from 312 to 48 bytes (44 for puffs & blood). This is still a lot of data for 1 SSG shot (~900 bytes), and we can probably pull it down even more if we get smarter (puffs don’t need angle, pitch or flags, can save 16 bytes, or 320 per SSG shot, there).

Rev. c6a8550 is my latest attempt at solidifying clientside sync (along with a couple of preceding commits). After trying dozens of different schemes, I finally discovered one that works nearly perfectly.

First of all, clients need to know what commands a server ran between the delta’s start state and end state. This isn’t so important for consoleplayer’s position, but it important in order to get accurate sounds from the other players. The client runs commands from other players (non-consoleplayer commands, or “NCP” commands), during this phase, called “synchronization”, and that’s where the it will start sounds such as rockets/plasma firing, lifts starting, etc. It’s possible for consoleplayer’s commands to affect NCPs’ positions, so they need to run in tandem. Currently they are run all at once; it would be better to have a way to run them the way the server did, or at least stretch them out as much as possible.

Once this is finished, the client loads the new, latest state received from the server. Re-running the commands the server ran between the delta’s start & end state is an imperfect process, so it can’t be relied on to perfectly create the state. After this step, synchronization is complete, and re-prediction commences.

The server won’t always run consoleplayer’s commands in lock-step. The client may send commands ¹⁰⁰⁄₈₀, ¹⁰¹⁄₈₁, and ¹⁰²⁄₈₂ (TIC/Index), but the server may run them all on TIC 101. Let’s stipulate that command 81 triggered a lift at TIC 101.

The client will then (possibly, but let’s say it does for the sake of our example) receive a delta “server received commands <= 82 at TIC 101”. So it synchronizes all commands <= 82, and spawns the lift at 101.

When the client first ran command 82, the lift had already been activated, so this TIC (102) would be the 2nd time the lift had been moved. However, after synchronization, the TICs will still align, but the commands won’t. Let’s say that the client has 3 commands left (¹⁰³⁄₈₃, ¹⁰⁴⁄₈₄, ¹⁰⁵⁄₈₅) that the server has yet to acknowledge receipt of. It will now run each of these commands separately, and move the lift 3 more times. However, initially, by the time the client ran command 85, the lift had already been moved 5 times. So when the client initially predicted the world out to ¹⁰⁵⁄₈₅, the lift had moved 5 times, but now after re-prediction, the lift has only moved 4 times. This will cause the lift to snap back 1 TIC’s worth of movement.

You’ll also notice that the client is about to run TIC 102, but will then run command ¹⁰³⁄₈₃. This disparity allows the client to know that it’s fallen behind. When the client notices this, it runs an extra TIC, without a command. This runs the world simulation, but doesn’t move consoleplayer.

Ever since TotalTrash was converted to a static site, I’ve wanted a way to add pages without copying and pasting tons of markup and updating the navbar. I’ve finally built a new site using Hugo, a static site generator written in Go.

Additionally, I plan to feature more news on TT. I’ve wanted a place to post D2K updates, and I think TT is where that belongs.

I may backdate some posts just for content’s sake. We’ll see how it goes.

I’ve decided to start working on a new source port called Doom2K or “D2K” for short. My goal is to build a state-of-the-art multiplayer source port capable of being the standard-bearer for multiplayer Doom. This, in my mind, means a couple of things.

Message-Based Netcode Is Broken

ZDaemon, Zandronum, and Odamex all use message-based netcode. For example, there is (historically) a function called P_SpawnMobj which spawns a new actor into the game. Serverside, inside of P_SpawnMobj is a call to a function that broadcasts a “mobj spawned” message; let’s call it SV_BroadcastMobjSpawned. For illustration’s sake, it generally looks something like this:

---------
| ID    |
---------
| X     |
---------
| Y     |
---------
| Z     |
---------
| Angle |
---------
| Type  |
---------

This model “works” for any number of game events. EECS, for example, has 44 messages. Odamex has 112. The way message-based netcode “scales up” is message proliferation, and there are a number of downsides to this.

You have to litter your code with network messages

Ugly, error-prone, and overly complicates extending the engine (scripting).

You have to exempt huge swaths of code from running clientside

Same as above.

Consistency is complicated

Unless you package up all your message into a single packet (assuming it will all fit), consistency requires sequencing and a “tic ended” message.

Doom Scripting Is Busted

Probably the best scripting out there is Doomsday. Otherwise, you can use EDGE/3DGE and COAL, or you can use ZDoom and ACS. I think you can use FraggleScript in Legacy.

The situation isn’t great though.

With a powerful scripting language, it would be possible to implement new game modes (like CTF), convert most of Doom’s physics and original game play, and represent assets and configuration in scripting also. PWO, for example, could simply be a user-provided function in a config.

ASCII Is Out, Unicode Is In

Unless Doomsday uses UTF-8, I’m unaware of any source ports that are Unicode-aware. This has serious implications for Doom’s popularity outside of English-speaking countries.

Software Rendering Is Limited And Slow

Software rendering is much slower than hardware-accelerated OpenGL rendering. In general, people who prefer software rendering do so because it’s more faithful to Doom’s original renderer (which almost no port uses anymore, due to its bugs and limitations). However, with display pixel count rising, software rendering is becoming less and less feasible.

Software renderers also have various limitations: lack of support for portals, 3D architecture (slopes, room-over-room, etc.), bad math (wobbling flats, problems with long walls, inaccurate flat rendering, etc.), and so on.

Goals

My major goals are to fix all of this.

Instead of network messages, D2K will use delta-compressed game states, which avoids all of the problems with message-based netcode (at the expense of increased CPU usage and RAM consumption).

D2K will use Lua for scripting. Current plans include:

• Implement console in scripting
• Implement new game modes in scripting
• Move Doom physics, game play, and assets to scripting

D2K will use UTF-8 instead of ASCII.

Regarding the renderer, I haven’t entirely decided to remove the software renderer. My main focus will be on the OpenGL renderer, however, because it’s easier to add advanced renderer features to it. Further, I will work on implementing a shader for the OpenGL renderer that accurately emulates Doom’s lighting.