Haven't updated in a while, so I thought I might write a post to give a little update on what's happening with Tengoku.
- If you've taken a look at the repository you might have noticed that a lot of change happened in the organization of the files; I ditched the colorfully named sub-libraries and cleaned out a lot of junk, including most of the bad, inaccurate docs lying around, old tests etc. The tests folder is very nice for doing experiments that double as tests.
- I finally settled on PascalCase for the entire thing and have been slowly updating the source code. I know I had just wrote that I decided that allcaps was the ultimate but then I read some code written by Pablo Reda implementing an IMGUI and found it to be very nice to read code with a mix of case, even in Forth, so decided that it was worth it. SwiftForth is case-insensitive so I haven't made the source perfectly consistent in regards to case but it definitely made it much more readable. I used to do all lower case mostly out of laziness, and still do when the code is throwaway. But hyphenation is an old Forth convention that I don't really like so I've started weeding that out - I still plan to use it though when I want to really distinguish the first word from the rest of the name of a function.
- New features are being added and changes are being made to the OOP framework all the time. I recently added two new flavors of classes to complement the Module keyword: Structure and Topic. These are cut-down versions of Module that don't support defining methods, to save memory, and to make intention clearer. Topic is meant to not have any properties or methods at all, only internal functions and static members. It's basically a convenient way of defining and working with a private wordlist that works with the same semantics as other classes.
- You can now open any class's internal words with the words [Using and Using], but I decided that I hated typing that so then I added [u and u] which is easier to type and u: which lets you do just one word, and then I recently checked up on Retro and saw its version, so I decided to experiment with the syntax "With: ... Without" . So there are three semantics for the same thing because I still haven't decided. Probably [u ... u] / u: in compile mode and With: ... Without in interpret mode.
- Inspection has been beefed up a lot and you can peek into all sorts of objects and get useful info. It's very nice to be able to quickly see what's going on in a group of variables - sort of a poor man's Visual Studio but nicer because you control exactly what the computer is showing you at any time. It's not hard, just put an object on the stack and add M?. They can be easily thrown into a dynamic textbox to have a live running update over-top the game. .S was enhanced at some point to by some black magic know what items on the stack are modules. Very nice and will be integrated into the HUD console I'm planning to add - that's a very old thing that I've had in every game engine experiment I've done since 2000 - but I found SwiftForth's sufficient up til this point. It will be very very cool, I've got lots of great ideas for it, such as a Fixed-point stack view and more like live-updated Current and Context state.
- I plan to enhance .Fields to show you summaries of non-classed fields based on their size ( if more than 4, judge from there - 8 would mean a double, more would mean an array ... of course it won't show the entire contents of a huge field, maybe just a size and the first and last bytes).
- I'm going to add a new feature that will work very nicely in Forth and will make certain pieces of OO code look like normal procedural code. It's called Property Sharing. You'll be able to define certain properties (I call them fields, but whatever) in a module as being Shared , and then in another module you'll be able to say that you want to share the properties of another module using a connection that you give a name to. Then you'll be able to invoke those fields without qualifying them at all, but they won't implicitly refer to This, instead they refer to the object that This (or the object on the stack, for external facing fields) is connected to. If you do qualify them, they will point to the properties not necessarily of that object but the one it is connected to. The connection is a plain old var that you must assign for the sharing to work. Of course, objects with shared fields automatically define their own connection to themselves. This will be used to streamline Entities and Components (currently the same thing is being accomplished with some ugly custom code), and make applications that utilize many interdependent objects.
- Now Tengoku has its own windowing and input code, called Iota. SDL 1.3 was too buggy and it wasn't being regularly updated so I moved to a library I just found out about called SFML. It's great, better than SDL, but it doesn't work on many computers with ATI cards. So I ported a small subset to Tengoku. Took me about 3 days, but, it works like a charm, and it can easily be swapped out for the real thing (when and if SFML matures to a level I feel that I could rely on), because I access it in the engine via a set of wrapper functions that match SFML. It's even compatible with SFML 1.6's data structures and events. Of course, the corresponding code was refactored heavily to make the task easier and came out to about ... 33% of the C++ code, I think. Iota was developed in a private repository that sat nested within the Tengoku folder structure in an ignored folder, so it could include some selected files that it needed, then it was migrated into and debugged using the various test programs.
- The GUI framework, called Studio, is fleshing out nicely. It is very easy to work with and you can do some fancy things like set up a game to support the mouse being able to pick up things and move them wherever you like.
I'm also doing some other things that I'm not sharing open-source. I'm developing the game, of course, but I don't want to talk about it. And I'm developing a set of tools built on the Studio (GUI) framework to work on graphics and maps. There's a comment in Studio.f that I forgot to take out that mentions ideas for this.
Guess that's about enough for today. I'm just glad the system is stable and mature enough now that I feel I can do anything with it.
Monday, December 27, 2010
Monday, December 6, 2010
Took my own advice and removed an inappropriate use of polymorphism. There's a somewhat technical word EDATA which points other words to the ENTITY object for any object that implements the IENTITY interface. (Components implement it.) It was originally a method. Methods are slow. But SO many words called EDATA, crucial stuff like position. I changed EDATA to a code word; now all it does is fetch the address pointed to by THIS; the rule is, all IENTITY's have to store a pointer to an ENTITY in the first cell. Bind by memory layout instead of method table, get 35% speed gain.
Only Forth lets you break out of structure to make better use of resources so elegantly and painlessly. I think it's a preview into a future of less waste.
Only Forth lets you break out of structure to make better use of resources so elegantly and painlessly. I think it's a preview into a future of less waste.
Argh
OK, no, I'm not in good mood.
Turns out, spending all these months adding modern features to Forth has ended up making it harder to debug (and therefore write), more complicated, AND, slower. Well, maybe. It might be my fault for ab/using these new features. Not using them appropriately. I had a feeling about it, but I was so concerned about looking "legit" to the public, making the code all "high level" with lots of generics. I am not sure if anyone cares right now - which doesn't bother me at all, because it means I don't have to be "legit" to keep anyone's attention. Since no one is bothering me, and Plan A didn't lead to a big bang, I can take it wherever I want.
Turns out generics have a price. Turns out they complicate things if you don't use them appropriately; maybe one central polymorphism system is hugely unnecessary some of the time. Maybe I need to say OK here are places where it can make things easier (high level) and here is where it would actually be easier if I ignored them (low level).
Forth is assembly on steroids. It makes low level seem high level, makes it fit in the palm of your hand, makes it friendly; manageable. Forth likes dumb data types, and keeping things flat - not building systems upon systems, but instead making specialized systems out of bits and pieces - that's flatter design that doesn't necessarily hinder the complexity of your ideas but keeps them easier to test and change. If those low level bits and pieces are reliable and rich, starting "from scratch" should be less painful.
Admitedly, some systems are complex. But when you admit that, the fact that complex systems are hard to implement becomes a great motivation to simplify; not necessarily to not do your idea, but to let Forth help you do it in a dumb, flat, manageable way. If you're smart and confident about Forth, you'll get that working and then implement the complex stuff under the hood, invisible and irrelevent to the calling code.
And you shouldn't proliferate redundant datatypes (integers, fixed point, halfword, bytes), unless you really REALLY need to for some reason. Polymorphism helps manage that (and Forth hates lots of types since it has no typing system), but it has to be appropriate. For instance I know that it's prohibitive for building dynamic vertex lists, because each call to write out a vertex would be twice as expensive. But for strings, it makes sense because it's an inherently non-realtime thing that you do in short bursts. Polymorphism would help solve the problem of having to manipulate strings for external API's that love them, or for storing XML, or writing out Forth code :)
Forth has wordlists, a concept that lines up neatly with modular program design. I'm using them, but maybe not as well as I could be. Because sometimes I use polymorphism when reaching into a wordlist would be fine, and more efficient (direct call instead of a method call).
What's so weird is when everyone else says that making a program fast means "hard to read" or "complex", the old collision grid code was soooo much faster yet it was about the same size it is now. Less if you count the assembly code I had to write to make it reasonable.
I get hung up on loading everything at once, maybe it's a laziness; as if to say "you never know when something will break" so you compile everything in case something does. That is distracting; the human brain wasn't designed to work on more than one thing at once, because when it does, when I'm able to concentrate on one thing at a time, I get a lot more efficient. If I used wordlists more often I'd be forced to narrow focus because it's inconvenient to be constantly modifying the search order.
OK, so, what have we learned ... it's good to break it down, figure out what you've been doing in the broad scope, and what you're doing right now, and make the bits and pieces ("primitives") support that by being fast and easy to use. Perhaps wrap the polymorphic system AROUND primitives (same name, separate wordlist(s)) to virtualize them for higher level systems; but only when it's appropriate; probably great for modular plugin things.
Turns out, spending all these months adding modern features to Forth has ended up making it harder to debug (and therefore write), more complicated, AND, slower. Well, maybe. It might be my fault for ab/using these new features. Not using them appropriately. I had a feeling about it, but I was so concerned about looking "legit" to the public, making the code all "high level" with lots of generics. I am not sure if anyone cares right now - which doesn't bother me at all, because it means I don't have to be "legit" to keep anyone's attention. Since no one is bothering me, and Plan A didn't lead to a big bang, I can take it wherever I want.
Turns out generics have a price. Turns out they complicate things if you don't use them appropriately; maybe one central polymorphism system is hugely unnecessary some of the time. Maybe I need to say OK here are places where it can make things easier (high level) and here is where it would actually be easier if I ignored them (low level).
Forth is assembly on steroids. It makes low level seem high level, makes it fit in the palm of your hand, makes it friendly; manageable. Forth likes dumb data types, and keeping things flat - not building systems upon systems, but instead making specialized systems out of bits and pieces - that's flatter design that doesn't necessarily hinder the complexity of your ideas but keeps them easier to test and change. If those low level bits and pieces are reliable and rich, starting "from scratch" should be less painful.
Admitedly, some systems are complex. But when you admit that, the fact that complex systems are hard to implement becomes a great motivation to simplify; not necessarily to not do your idea, but to let Forth help you do it in a dumb, flat, manageable way. If you're smart and confident about Forth, you'll get that working and then implement the complex stuff under the hood, invisible and irrelevent to the calling code.
And you shouldn't proliferate redundant datatypes (integers, fixed point, halfword, bytes), unless you really REALLY need to for some reason. Polymorphism helps manage that (and Forth hates lots of types since it has no typing system), but it has to be appropriate. For instance I know that it's prohibitive for building dynamic vertex lists, because each call to write out a vertex would be twice as expensive. But for strings, it makes sense because it's an inherently non-realtime thing that you do in short bursts. Polymorphism would help solve the problem of having to manipulate strings for external API's that love them, or for storing XML, or writing out Forth code :)
Forth has wordlists, a concept that lines up neatly with modular program design. I'm using them, but maybe not as well as I could be. Because sometimes I use polymorphism when reaching into a wordlist would be fine, and more efficient (direct call instead of a method call).
What's so weird is when everyone else says that making a program fast means "hard to read" or "complex", the old collision grid code was soooo much faster yet it was about the same size it is now. Less if you count the assembly code I had to write to make it reasonable.
I get hung up on loading everything at once, maybe it's a laziness; as if to say "you never know when something will break" so you compile everything in case something does. That is distracting; the human brain wasn't designed to work on more than one thing at once, because when it does, when I'm able to concentrate on one thing at a time, I get a lot more efficient. If I used wordlists more often I'd be forced to narrow focus because it's inconvenient to be constantly modifying the search order.
OK, so, what have we learned ... it's good to break it down, figure out what you've been doing in the broad scope, and what you're doing right now, and make the bits and pieces ("primitives") support that by being fast and easy to use. Perhaps wrap the polymorphic system AROUND primitives (same name, separate wordlist(s)) to virtualize them for higher level systems; but only when it's appropriate; probably great for modular plugin things.
Optimizing: Global Collision System
Over the weekend, after working on the Studio application a bit, I got a little obsessed over the global collision system. Just a little bit of background; this is the system for partitioning the game's space into a grid of sectors and so culling down the number of objects it needs to test for overlap, using simple hitboxes. Consider it the broad collision phase. Also, this code was written for an engine from a few years ago, when I didn't use formal object orientation, or polymorphism, or components. As a result it was very fast - on a 1.7ghz machine, it could do 5000x5000 object tests at 60fps (and drawing ALL 5000 of them using Allegro, with CPU to spare).
After a first run of adaptation for Tengoku, the thing's performance was dismal; it couldn't do 5000x5000 at all, on a 3.3 ghz computer. (The old version also couldn't do objects bigger than 256x256, but the recent fixing of that is totally not the reason for the slowdown.) So, I felt forced to start optimizing to get it usable ... and here's what I did:
After a first run of adaptation for Tengoku, the thing's performance was dismal; it couldn't do 5000x5000 at all, on a 3.3 ghz computer. (The old version also couldn't do objects bigger than 256x256, but the recent fixing of that is totally not the reason for the slowdown.) So, I felt forced to start optimizing to get it usable ... and here's what I did:
- Sped up LIST:ADDTO; all it really needed to do was store the given thing into the last cell and increment the length field.
- Sped up LIST:EACH ; for some reason using an older, longer version of the routine sped things up just itterating objects by around 40%.
- Extended ENTITY (which affects memory usage globally ... so I guess that now I'm specializing for 2D...)
- Cached absolute rectangles for each object.
- Added BOX@ and used that instead of @BOX. (a dumbly named word) which leads to an assumption that the HITRECT attributes point to BOXes (which kind of defeats the purpose of IRECTANGLE ...)
- Sped up WORDMAP:LOOKUP by turning it into a CODE word; 2 hours. :(
All that work and eventually, performance was significantly improved. 1500x1500 objects in 7ms, when at first it couldn't do that many at all (it led to a spiral of death). It's still not good enough, in my opinion. I keep thinking something is wrong. I might do an more intense investigation into exactly what is happening; how many objects each object actually checks, how long each check takes...
But anyway, for the last thing I wanted to write about, I learned something amazing, and somewhat troubling. That the following piece of code turns a loop that took 1.5 ms into 6.5 ms!!!!
: ASDF DROP ;
: FDSA ['] ASDF MYLIST EACH ;
: FDSA ['] ASDF MYLIST EACH ;
Why? The only conclusion I can take from this is that modern, prefetching, superscalar processors hate, detest, and abhore very short subroutines. Granted most of my functions aren't that short but some get pretty close! It worries me because sometimes it has a good reason, such as the virtualizing mechanism that lets any component be treated as it's owner, where you have routines like this:
: POS EDATA >_POS ;
Which comes out to a CALL and an ADD instruction. Seems to me that this MUST MUST MUST be inlined. I'll try that out when I get home.
Subscribe to:
Posts (Atom)