Some updates

Haven't updated in a while, so I thought I might write a post to give a little update on what's happening with Tengoku.

- If you've taken a look at the repository you might have noticed that a lot of change happened in the organization of the files; I ditched the colorfully named sub-libraries and cleaned out a lot of junk, including most of the bad, inaccurate docs lying around, old tests etc.  The tests folder is very nice for doing experiments that double as tests.

- I finally settled on PascalCase for the entire thing and have been slowly updating the source code.  I know I had just wrote that I decided that allcaps was the ultimate but then I read some code written by Pablo Reda implementing an IMGUI and found it to be very nice to read code with a mix of case, even in Forth, so decided that it was worth it.  SwiftForth is case-insensitive so I haven't made the source perfectly consistent in regards to case but it definitely made it much more readable.  I used to do all lower case mostly out of laziness, and still do when the code is throwaway.  But hyphenation is an old Forth convention that I don't really like so I've started weeding that out - I still plan to use it though when I want to really distinguish the first word from the rest of the name of a function.

- New features are being added and changes are being made to the OOP framework all the time.  I recently added two new flavors of classes to complement the Module keyword: Structure and Topic.  These are cut-down versions of Module that don't support defining methods, to save memory, and to make intention clearer. Topic is meant to not have any properties or methods at all, only internal functions and static members.  It's basically a convenient way of defining and working with a private wordlist that works with the same semantics as other classes.

- You can now open any class's internal words with the words [Using and Using], but I decided that I hated typing that so then I added [u and u] which is easier to type and u: which lets you do just one word, and then I recently checked up on Retro and saw its version, so I decided to experiment with the syntax "With: ... Without" .  So there are three semantics for the same thing because I still haven't decided.  Probably [u ... u] / u: in compile mode and With: ... Without in interpret mode.

- Inspection has been beefed up a lot and you can peek into all sorts of objects and get useful info.  It's very nice to be able to quickly see what's going on in a group of variables - sort of a poor man's Visual Studio but nicer because you control exactly what the computer is showing you at any time.  It's not hard, just put an object on the stack and add M?.   They can be easily thrown into a dynamic textbox to have a live running update over-top the game.   .S was enhanced at some point to by some black magic know what items on the stack are modules.  Very nice and will be integrated into the HUD console I'm planning to add - that's a very old thing that I've had in every game engine experiment I've done since 2000 - but I found SwiftForth's sufficient up til this point.  It will be very very cool, I've got lots of great ideas for it, such as a Fixed-point stack view and more like live-updated Current and Context state.

- I plan to enhance .Fields to show you summaries of non-classed fields based on their size ( if more than 4, judge from there - 8 would mean a double, more would mean an array ... of course it won't show the entire contents of a huge field, maybe just a size and the first and last bytes).

- I'm going to add a new feature that will work very nicely in Forth and will make certain pieces of OO code look like normal procedural code.  It's called Property Sharing.  You'll be able to define certain properties (I call them fields, but whatever) in a module as being Shared , and then in another module you'll be able to say that you want to share the properties of another module using a connection that you give a name to.  Then you'll be able to invoke those fields without qualifying them at all, but they won't implicitly refer to This, instead they refer to the object that This (or the object on the stack, for external facing fields) is connected to.  If you do qualify them, they will point to the properties not necessarily of that object but the one it is connected to.  The connection is a plain old var that you must assign for the sharing to work.  Of course, objects with shared fields automatically define their own connection to themselves.  This will be used to streamline Entities and Components (currently the same thing is being accomplished with some ugly custom code), and make applications that utilize many interdependent objects.

- Now Tengoku has its own windowing and input code, called Iota.  SDL 1.3 was too buggy and it wasn't being regularly updated so I moved to a library I just found out about called SFML.  It's great, better than SDL, but it doesn't work on many computers with ATI cards.  So I ported a small subset to Tengoku.  Took me about 3 days, but, it works like a charm, and it can easily be swapped out for the real thing (when and if SFML matures to a level I feel that I could rely on), because I access it in the engine via a set of wrapper functions that match SFML.  It's even compatible with SFML 1.6's data structures and events.  Of course, the corresponding code was refactored heavily to make the task easier and came out to about ... 33% of the C++ code, I think.  Iota was developed in a private repository that sat nested within the Tengoku folder structure in an ignored folder, so it could include some selected files that it needed, then it was migrated into and debugged using the various test programs.

- The GUI framework, called Studio, is fleshing out nicely.  It is very easy to work with and you can do some fancy things like set up a game to support the mouse being able to pick up things and move them wherever you like.

I'm also doing some other things that I'm not sharing open-source.  I'm developing the game, of course, but I don't want to talk about it.  And I'm developing a set of tools built on the Studio (GUI) framework to work on graphics and maps.  There's a comment in Studio.f that I forgot to take out that mentions ideas for this.   

Guess that's about enough for today.  I'm just glad the system is stable and mature enough now that I feel I can do anything with it.

Took my own advice and removed an inappropriate use of polymorphism.  There's a somewhat technical word  EDATA which points other words to the ENTITY object for any object that implements the IENTITY interface.  (Components implement it.)  It was originally a method.  Methods are slow.  But SO many words called EDATA, crucial stuff like position.   I changed EDATA to a code word; now all it does is fetch the address pointed to by THIS; the rule is, all IENTITY's have to store a pointer to an ENTITY in the first cell.  Bind by memory layout instead of method table, get 35% speed gain.

Only Forth lets you break out of structure to make better use of resources so elegantly and painlessly.  I think it's a preview into a future of less waste.

Argh

OK, no, I'm not in good mood.

Turns out, spending all these months adding modern features to Forth has ended up making it harder to debug (and therefore write), more complicated, AND, slower.    Well, maybe.  It might be my fault for ab/using these new features.  Not using them appropriately.  I had a feeling about it, but I was so concerned about looking "legit" to the public, making the code all "high level" with lots of generics.  I am not sure if anyone cares right now - which doesn't bother me at all, because it means I don't have to be "legit" to keep anyone's attention.  Since no one is bothering me, and Plan A didn't lead to a big bang, I can take it wherever I want.

Turns out generics have a price.  Turns out they complicate things if you don't use them appropriately; maybe one central polymorphism system is hugely unnecessary some of the time.  Maybe I need to say OK here are places where it can make things easier (high level) and here is where it would actually be easier if I ignored them (low level).

Forth is assembly on steroids.  It makes low level seem high level, makes it fit in the palm of your hand, makes it friendly;  manageable.  Forth likes dumb data types, and keeping things flat - not building systems upon systems, but instead making specialized systems out of bits and pieces - that's flatter design  that doesn't necessarily hinder the complexity of your ideas but keeps them easier to test and change.   If those low level bits and pieces are reliable and rich, starting "from scratch" should be less painful.

Admitedly, some systems are complex.  But when you admit that, the fact that complex systems are hard to implement becomes a great motivation to simplify; not necessarily to not do your idea, but to let Forth help you do it in a dumb, flat, manageable way.  If you're smart and confident about Forth, you'll get that working and then implement the complex stuff under the hood, invisible and irrelevent to the calling code.

And you shouldn't proliferate redundant datatypes (integers, fixed point, halfword, bytes), unless you really REALLY need to for some reason. Polymorphism helps manage that (and Forth hates lots of types since it has no typing system), but it has to be appropriate.  For instance I know that it's prohibitive for building dynamic vertex lists, because each call to write out a vertex would be twice as expensive.  But for strings, it makes sense because it's an inherently non-realtime thing that you do in short bursts.  Polymorphism would help solve the problem of having to manipulate strings for external API's that love them, or for storing XML, or writing out Forth code :)

Forth has wordlists, a concept that lines up neatly with modular program design.  I'm using them, but maybe not as well as I could be.  Because sometimes I use polymorphism when reaching into a wordlist would be fine, and more efficient (direct call instead of a method call).

What's so weird is when everyone else says that making a program fast means "hard to read" or "complex", the old collision grid code was soooo much faster yet it was about the same size it is now.  Less if you count the assembly code I had to write to make it reasonable.

I get hung up on loading everything at once, maybe it's a laziness; as if to say "you never know when something will break" so you compile everything in case something does.  That is distracting; the human brain wasn't designed to work on more than one thing at once, because when it does, when I'm able to concentrate on one thing at a time, I get a lot more efficient.  If I used wordlists more often I'd be forced to narrow focus because it's inconvenient to be constantly modifying the search order.

OK, so, what have we learned ... it's good to break it down, figure out what you've been doing in the broad scope, and what you're doing right now, and make the bits and pieces ("primitives") support that by being fast and easy to use.  Perhaps wrap the polymorphic system AROUND primitives (same name, separate wordlist(s)) to virtualize them for higher level systems; but only when it's appropriate; probably great for modular plugin things.

Optimizing: Global Collision System

Over the weekend, after working on the Studio application a bit, I got a little obsessed over the global collision system.  Just a little bit of background; this is the system for partitioning the game's space into a grid of sectors and so culling down the number of objects it needs to test for overlap, using simple hitboxes.  Consider it the broad collision phase.  Also, this code was written for an engine from a few years ago, when I didn't use formal object orientation, or polymorphism, or components.  As a result it was very fast - on a 1.7ghz machine, it could do 5000x5000 object tests at 60fps (and drawing ALL 5000 of them using Allegro, with CPU to spare).

After a first run of adaptation for Tengoku, the thing's performance was dismal; it couldn't do 5000x5000 at all, on a 3.3 ghz computer.   (The old version also couldn't do objects bigger than 256x256, but the recent fixing of that is totally not the reason for the slowdown.)  So, I felt forced to start optimizing to get it usable ... and here's what I did:

• Sped up LIST:ADDTO; all it really needed to do was store the given thing into the last cell and increment the length field.
• Sped up LIST:EACH ; for some reason using an older, longer version of the routine sped things up just itterating objects by around 40%.
• Extended ENTITY (which affects memory usage globally ... so I guess that now I'm specializing for 2D...)
• Cached absolute rectangles for each object.
• Added BOX@ and used that instead of @BOX. (a dumbly named word) which leads to an assumption that the HITRECT attributes point to BOXes (which kind of defeats the purpose of  IRECTANGLE ...)
• Sped up WORDMAP:LOOKUP by turning it into a CODE word; 2 hours. :(

All that work and eventually, performance was significantly improved.  1500x1500 objects in 7ms, when at first it couldn't do that many at all (it led to a spiral of death).  It's still not good enough, in my opinion.  I keep thinking something is wrong.  I might do an more intense investigation into exactly what is happening; how many objects each object actually checks, how long each check takes... 

But anyway, for the last thing I wanted to write about, I learned something amazing, and somewhat troubling.  That the following piece of code turns a loop that took 1.5 ms into 6.5 ms!!!!

: ASDF   DROP ;
: FDSA   ['] ASDF MYLIST EACH ;

Why?  The only conclusion I can take from this is that modern, prefetching, superscalar processors hate, detest, and abhore very short subroutines.  Granted most of my functions aren't that short but some get pretty close!  It worries me because sometimes it has a good reason, such as the virtualizing mechanism that lets any component be treated as it's owner, where you have routines like this:

: POS   EDATA >_POS ;

Which comes out to a CALL and an ADD instruction.  Seems to me that this MUST MUST MUST be inlined.  I'll try that out when I get home.

I started using and converting the source to uppercase.  I don't know maybe I'm crazy but it seems to make me more aware of how many words I'm using.  And it helps differentiate between stuff that's Forth code and stuff that's not.  (comments, external functions)

Speedups

Recently I optimized several parts of the SDK, mostly to be able to support a "bullet hell" type game (one of the ideas I'm kicking around.)

Here is a little summary of what I accomplished:

• Entities are much faster
• The heap is now faster and more memory efficient and lets you allocate up to 256MB 16MB (up from 1MB!)
• The Layer component was eliminated and folded into Entity so parenting is more efficient.
• Spritebatches were implemented; performance is not too shabby with 5000 moving sprites a frame on a 3.2ghz desktop.  It's definitely CPU bottlenecked; eventually I'll offload sprite rendering completely to the GPU via....
• The Shader module; shaders will bring huge performance gains without a doubt :P 

Providing Tengoku as a DLL

Today I considered that to make Tengoku accessible from any language, practically all you need is the system compiled as a dynamic library, and this API of 4 whopping functions. I think.

• ten_evaluate ( string ) : string
• ten_push ( int ) : void
• ten_pop () : int
• ten_wrap ( function ) : int

And maybe double and triple variants for ten_push and ten_pop.

Ten_wrap would take a native function and make it something that Tengoku can execute, since it makes extensive use of callbacks. 

Ten_evaluate would take a simple string containing Forth "script", likely usually just a single function name, and return a string of whatever was output to the console.  Since it would be against SwiftForth's commercial license, all code compilation words would be inaccessible; and you could only access game engine words.  (So, all logic would be in your native language, which would be the point of it anyway.)

On second thought, that probably isn't it.  You also need multitasking and messages, which right now are intrinsic to Plexi and Walnut, respectively.  And a more efficient way of getting the address of objects and then their properties, else it would just be too slow to be more than a demo.

Maybe there can be a way of automatically go through and generate the DLL and headers using the stack comments. 

Starting to document things

http://code.google.com/p/tengoku-engine/wiki/Plexi
http://code.google.com/p/tengoku-engine/wiki/Jump

It's really slapdash, and I am not documenting every little thing.  Just the stuff I need to have in my head or check.

Update - abolishing ->

In the midst of a cleanup.  Decided that I don't have a use for a revamped module internals wordset yet, instead I added a couple needed features. 

1) Now you can jump to a method implementation in the source using the convention ":".  

2) A new kind of method, the "xmethod", or "external method", methods intended to face outside, complementing their non-method counterparts.  Work like normal methods but always requires the module to be on the stack.  I'm in the process of abolishing "->", so this is one step towards that goal.  The rule is, if a method is generally meant to be an interface to the OUTSIDE, make it external.  If it is used internally only, make it a normal one.  Once -> is gone though, { } clauses or AS will be the only ways to use normal methods.

3) Because you occasionally have to call external methods from a { } clause, I added a shorthand for THIS, the dollar sign ("$").

1.41.3.0 Build

What's new in this one is mainly the global sprite collision system.

Posted on the Downloads page.

Tidbits

I recently added the last feature of global bounding box collisions, making all the so-far planned features complete.

The interesting bit is this; I found that Plexi's module-related punctuation ( { }, -> ) made programming too un-Forth-like, and  created encumbrances in the system due to the one-size-fits-all nature.  My solution; let the user customize the system for every problem.  This is the way Forth does it from the ground up; I think that by keeping the module data structures as the "lingua franca" but allowing easy access to its internals will open up novel ways of using them that goes beyond what traditional OO systems let you do.

Project page update

I updated the copy on the project page

http://code.google.com/p/tengoku-engine/

Possibly transforming refactoring looming

I plan on taking the features of the module system, and "exploding" them.  That's a term I just came up with.  The best analogy is probably like toppling the tower of Babel.  Everything you need still exists, it's just going to be even MORE horizontal.  Actually in this case it's more like demolishing a 2-story house and making it more like a rancher.  Or a big basketball court.  OK I'm done with analogy-making tonight.

Fixed zip file

Uploaded a new build on the project page, thanks to William Yates for pointing out to me that the zip file was corrupt!

I also fixed an elusive problem with multitasking and the component system.

Grab it from Downloads.

1.40.4.0 Build

This build is pretty stable. I've recently added a form of multiple inheritance for modules. This includes the fixes for XP.

http://code.google.com/p/tengoku-engine/downloads/list

Full list of changes after the jump.

Early Game Build

I recently got .EXE-export working.

This is a very early build of a game I'm working on.

http://rogerlevy.net/games/kevin/bin.zip

DLL Issue Resolved?

I think that this problem has been fixed.  There was a DLL required by Allegro that was missing on presumably every computer but my Windows 7 desktop.  It doesn't make any sense to me, because I didn't know Allegro required this DLL, but I've already put it in the dev tree so it'll be in the next release.

DLL Issues

Due to what I think is a difference between the way DLL's are loaded by Windows 7 and Windows XP, the SDK doesn't run properly in XP.  I have gotten it to work in one or the other, but not both.  Will work on this before releasing the next build ...

1.29.12.0 Build

Hey there!  Thanks for checking out the Tengoku Blog.  This is where I'll be posting updates, and they should be more or less weekly, so why not subscribe to the feed, or click the follow button?

I want to announce a  new alpha build that is up on the project page.  Click to go to the download page and download it:  http://code.google.com/p/tengoku-engine/downloads/list

See docs/changes.txt for what's new.

This revision is a big one.  Lots of bugs were fixed, and lots of essential features were added.  The previous alpha builds were "source code proofs" more than anything, and I had no real idea how stable things were.  Now I am starting to see Tengoku become solid, although it has become a little more complicated than I expected.  The gametest_02 contains an odd little interactive example / CPU stress test.  All of the other apps are currently broken.

If you were having problems trying to see the library in action, running gametest_02 *should* work for you this time - I hope.

If you find any bugs (or have trouble running anything), PLEASE, let me know!  I have a limited ability to find everything on my own and I'll appreciate any feedback, even mean/bitchy ones.  (I'll continue developing, since I know this engine will be useful to me no matter what.)  Kind ones are cool too, though!  ;P