Saturday, February 5, 2011
Tuesday, February 1, 2011
On documenting the engine
I decided that it would be a good idea to not document the engine with the widespread approach of simply documenting the API, and explaining things piecemeal. Almost always that stuff is generated from big clunky interleaved function comments that dwarf the code itself and are really painful to read. There are two short-term reasons I don't want to do it this way.
- The text editor TextPad can't handle separate docs, I am working with dumb flat text files for the time being, and I really don't want to clutter things up with interleaved doc comments.
- I never liked API documentation. They focus on what functions do, and not how to use them or what they mean in the big picture.
Instead, I think that this will be my approach for now ...
- Explain the framework in detail; this part doesn't change anymore, so this will be like the "key" to the whole rest of the system.
- Make the code more self-documenting, with stack comments everywhere except where it would be ridiculously obvious or obtrusive, and explain in comments only important things. Make formatting consistent so simple stack-comment-only glossaries could be generated automatically.
- Explain things from a bird's eye view in a way that describes the design so you know what you can do with it and where to find what you need.
- Make glossaries for interfaces. This part is relatively fixed.
- Make a cookbook of mini-tutorials for common stuff everyone would want to do. Deprecate/rewrite tutorials when/if API changes make them outdated.
Subject to revision, but I think I will give it a try and see how it goes.
Saturday, January 29, 2011
Added shared members
Finally got around to adding this feature which will eliminate an ugly hack in iEntity.f, and I think actually eliminate iEntity altogether? (Well, we'll see.)
Shared members allow one object to share properties with another one, so that they appear to be properties of the sharee, but they really are the sharer's.
The way it turned out to be implemented was surprisingly elegant.
To use it, you create a special property called a "reference" with the word "Via".
Example:
Module Entity
Via Owner
This makes a field called Owner for Entity that works like any other property. There is nothing special about it by itself. But now, all subsequent fields will become shared fields using that reference Owner.
Vector Make Pos
Var Visible
...
(And so on. To get out of that mode, you'd call Static, Embedded, or Global)
So now, instead of pointing to This like most fields, Pos and Visible would go through the address pointed to by Owner. To use them, you have to set Owner to something before doing anything. This might be made automatic for the sharer sometime, but for now, you have to set it manually in both constructors.
To share those fields, here's what you'd do:
Component MyComponent \ As an example, we're using a component, but you can do it with any Structure or Module too.
Using Entity Share Owner
Actually, Component will define its own Owner automatically, but for the sake of this example we're making it explicit.
First we call Using to add Entity's words to MyComponent's included classes. Then we say we want to share properties associated with the Owner reference. (What's kind of cute is that Share is nothing but an alias for Var.) So now, the Owner's Pos and Visible can be called from MyComponent's functions as if they were part of MyComponent. Even >Pos and >Visible work.
What's great about the implementation is references don't have to be properties of classes; they can be static, or even global. All the shared properties do is evaluate "Owner @" and then the property as normal. You can connect lots of interdependent objects without explicit coupling throughout the code; the connection is implicit and defined in one place.
Shared members allow one object to share properties with another one, so that they appear to be properties of the sharee, but they really are the sharer's.
The way it turned out to be implemented was surprisingly elegant.
To use it, you create a special property called a "reference" with the word "Via".
Example:
Module Entity
Via Owner
This makes a field called Owner for Entity that works like any other property. There is nothing special about it by itself. But now, all subsequent fields will become shared fields using that reference Owner.
Vector Make Pos
Var Visible
...
(And so on. To get out of that mode, you'd call Static, Embedded, or Global)
So now, instead of pointing to This like most fields, Pos and Visible would go through the address pointed to by Owner. To use them, you have to set Owner to something before doing anything. This might be made automatic for the sharer sometime, but for now, you have to set it manually in both constructors.
To share those fields, here's what you'd do:
Component MyComponent \ As an example, we're using a component, but you can do it with any Structure or Module too.
Using Entity Share Owner
Actually, Component will define its own Owner automatically, but for the sake of this example we're making it explicit.
First we call Using to add Entity's words to MyComponent's included classes. Then we say we want to share properties associated with the Owner reference. (What's kind of cute is that Share is nothing but an alias for Var.) So now, the Owner's Pos and Visible can be called from MyComponent's functions as if they were part of MyComponent. Even >Pos and >Visible work.
What's great about the implementation is references don't have to be properties of classes; they can be static, or even global. All the shared properties do is evaluate "Owner @" and then the property as normal. You can connect lots of interdependent objects without explicit coupling throughout the code; the connection is implicit and defined in one place.
Thursday, January 13, 2011
Shmup demo!
Here is a tech demo made out of one of the test programs I wrote for the shmup game.
A to shoot
Download
A to shoot
Download
Sunday, January 9, 2011
New OOP Syntax Details
From the Syntax2.f file:
Module
removes all modules from the search order
creates the structure, creates a wordlist, and sets the pointer to the size counter for field words
it doesn't add itself to the search order automatically
::
Compile a verb or method
If the word that follows is NOT a method ,
creates an internal word that can be accessed externally with a > prefix
both words it creates are private; to use them you have to add the module to the search order with Using
If the word that follows IS a method ,
it sets the following code to be the implementation for that method with this module.
and it creates a version using the Module:Method convention (called the direct version)
unlike methods, direct versions they are always internal.
Methods are still always public, including the direct versions.
Fields are always private now. They still have the internal and external versions, but you have to add the class to the search order with Using or :: to get to them.
Procedural
Remove all classes from the search order.
Var
now a shorthand for Variable but also supporting being a cell-size field
in a Module , fields are private.
in a Structure , fields are public.
Field and Make can also be used inside or outside a class.
Embedded
automatically called by Module (and Structure) , puts Var and friends
in "OOP" mode, i.e. they will be private or public depending on if
we're defining a module or a structure.
Make: and New
Automatic initialization; note this doesn't preclude the Class: convention
Make: ; is used to either embed a module into a class or
compile a module into the dictionary using *STATIC* parameters.
When a Module is instantiated, even if it's embedded
will be passed to that module's Init method.
'New' is the runtime version of the same principle. It passes whatever
parameters to the given Module's implementation of Init .
Module
removes all modules from the search order
creates the structure, creates a wordlist, and sets the pointer to the size counter for field words
it doesn't add itself to the search order automatically
::
Compile a verb or method
If the word that follows is NOT a method ,
creates an internal word that can be accessed externally with a > prefix
both words it creates are private; to use them you have to add the module to the search order with Using
If the word that follows IS a method ,
it sets the following code to be the implementation for that method with this module.
and it creates a version using the Module:Method convention (called the direct version)
unlike methods, direct versions they are always internal.
Methods are still always public, including the direct versions.
Fields are always private now. They still have the internal and external versions, but you have to add the class to the search order with Using or :: to get to them.
Procedural
Remove all classes from the search order.
Var
now a shorthand for Variable but also supporting being a cell-size field
in a Module , fields are private.
in a Structure , fields are public.
Field and Make can also be used inside or outside a class.
Embedded
automatically called by Module (and Structure) , puts Var and friends
in "OOP" mode, i.e. they will be private or public depending on if
we're defining a module or a structure.
Make: and New
Automatic initialization; note this doesn't preclude the Class: convention
Make:
compile a module into the dictionary using *STATIC* parameters.
When a Module is instantiated, even if it's embedded
will be passed to that module's Init method.
'New' is the runtime version of the same principle. It passes whatever
parameters to the given Module's implementation of Init .
Friday, January 7, 2011
Introducing an improved syntax for the OOP component
Introducing a improved syntax for the OOP component. It's a slight modification to what's currently in place, that'll hopefully be easier to work with and more logical. Trying to go for Zero-Context, that is, code whose location doesn't depend so much on the current state of the compiler. Hoping it'll encourage more interactive development, less recompilation from the beginning. Somewhat counter-intuitively, it locks everything but methods away privately to each class. However it is very easy to "include" them into the domains of the current class, or "open" them in what I call Procedural Mode, both using the Using keyword. I've adapted 3 classes to it and it seems to be working - but I'm about to try to playing with it some in the IDE to see if it fulfills its purpose.
Thursday, January 6, 2011
Great minds think alike.
TIL that this guy had proposed an object oriented extension to Forth that almost mirrors the one I designed for Tengoku. In 1996. That's 15 years ago now. :) Anyway it's so close it mirrors some of the semantics - Field , Method , and even { and } (which are sort of being phased out and replaced though by As , With , and the > prefix).
Link to the proposal document:
http://www.forth.org/literature/oopf.html
Also, I just found and will purchase this book: Object-Oriented Forth. Published in 1987!
http://search.barnesandnoble.com/books/product.aspx?r=1&r=1&IF=N&ISBN=0125635702&cm_mmc=Goodreads-_-k117601-_-j12871747k117601-_-Primary
On the coding front, I cleaned up the source code, and thinking I should probably do a little more. Going through the hedges with the clippers. :) Maybe a bit like Edward Scissorhands. :) It's really satisfying. Besides making it nice and pretty and clean, helps put the different components in perspective.
There is a lot of code and so much of it still shows residual signs of the confused, disorganized thinking process I was going through just to figure out what low level services worked well. I wish I had known at the beginning what I've accumulated through the process, but that's sort of a self contradicting statement.
I've got a pretty nice set of reliable services now, but there is just such a multitude of more stuff I want to do. Really made me realize how much progress has been made and continues to be made in the "Outside World". I'm playing a weird game of catchup. I want power, but I don't trust the tools that are available. Maybe now is time to stop "fortifying" for a moment and adopt a new groove of finishing products that add a few new features at a time.
The shmup performs pretty well on an aging 2005 lappy. 3000+ colliding bullets at 60hz with plenty of CPU to spare. Slowdown doesn't really impact smoothness much anyway though thanks to fixed timestep. It's implemented using a special module, the BulletMngr, that brings together a Spritebatch and a Collisiongrid and some other things but doesn't use entities because those are too slow for that many moving things. The majority of the time is spent drawing the bullets. A clever sprite-generating vertex shader would virtually eliminate any limit on sprite count. I like having the ability to have tons of sprites. The game isn't going to expect you to actually dodge 3000 though, exactly. :)
Really annoyed at the laptop though. It says it supports fragment (pixel) shaders but it actually doesn't! Witnessed this with 3dMark '06 and firsthand saw the driver spit back "can't link fragment shader" / "fragment shader not supported by HW" etc. So I can't develop post effects on it. :( The desktop works great though.
Actually, on the shader front, I think I did something that was really cool the other day - I implemented a floating-point framebuffer and an HDR shader that dithers the output, so you eliminate banding and can fake really, really subtle color degradations, especially in the lower luminance levels. It was very gratifying because here I've got the full spectrum of fidelity - I can do 8-bit pixely sprites (the default texture filtering is GL_NEAREST :) or I can render really smooth, anti-aliased geometry or bitmaps and do extremely smooth post effects with perceptually infinite color range. Isn't that nice? I think so.
Kind of afraid to just jump in and play with the engine still - it has trained me to think it could break at any moment - I think I'd rather work on the game idea so there is less unknown when I sit down to "implement" (in Forth implementing is ALWAYS experimentation though, ain't that nice ;). Inevitably I will want to redesign some feature or two or three to suit the game. But I think I've coded things in a modular enough way (hence putting almost everything, literally, in "modules") that I won't be running back to the starting line, and I have lots of reusable pieces that I really like as they are.
Link to the proposal document:
http://www.forth.org/literature/oopf.html
Also, I just found and will purchase this book: Object-Oriented Forth. Published in 1987!
http://search.barnesandnoble.com/books/product.aspx?r=1&r=1&IF=N&ISBN=0125635702&cm_mmc=Goodreads-_-k117601-_-j12871747k117601-_-Primary
On the coding front, I cleaned up the source code, and thinking I should probably do a little more. Going through the hedges with the clippers. :) Maybe a bit like Edward Scissorhands. :) It's really satisfying. Besides making it nice and pretty and clean, helps put the different components in perspective.
There is a lot of code and so much of it still shows residual signs of the confused, disorganized thinking process I was going through just to figure out what low level services worked well. I wish I had known at the beginning what I've accumulated through the process, but that's sort of a self contradicting statement.
I've got a pretty nice set of reliable services now, but there is just such a multitude of more stuff I want to do. Really made me realize how much progress has been made and continues to be made in the "Outside World". I'm playing a weird game of catchup. I want power, but I don't trust the tools that are available. Maybe now is time to stop "fortifying" for a moment and adopt a new groove of finishing products that add a few new features at a time.
The shmup performs pretty well on an aging 2005 lappy. 3000+ colliding bullets at 60hz with plenty of CPU to spare. Slowdown doesn't really impact smoothness much anyway though thanks to fixed timestep. It's implemented using a special module, the BulletMngr, that brings together a Spritebatch and a Collisiongrid and some other things but doesn't use entities because those are too slow for that many moving things. The majority of the time is spent drawing the bullets. A clever sprite-generating vertex shader would virtually eliminate any limit on sprite count. I like having the ability to have tons of sprites. The game isn't going to expect you to actually dodge 3000 though, exactly. :)
Really annoyed at the laptop though. It says it supports fragment (pixel) shaders but it actually doesn't! Witnessed this with 3dMark '06 and firsthand saw the driver spit back "can't link fragment shader" / "fragment shader not supported by HW" etc. So I can't develop post effects on it. :( The desktop works great though.
Actually, on the shader front, I think I did something that was really cool the other day - I implemented a floating-point framebuffer and an HDR shader that dithers the output, so you eliminate banding and can fake really, really subtle color degradations, especially in the lower luminance levels. It was very gratifying because here I've got the full spectrum of fidelity - I can do 8-bit pixely sprites (the default texture filtering is GL_NEAREST :) or I can render really smooth, anti-aliased geometry or bitmaps and do extremely smooth post effects with perceptually infinite color range. Isn't that nice? I think so.
Kind of afraid to just jump in and play with the engine still - it has trained me to think it could break at any moment - I think I'd rather work on the game idea so there is less unknown when I sit down to "implement" (in Forth implementing is ALWAYS experimentation though, ain't that nice ;). Inevitably I will want to redesign some feature or two or three to suit the game. But I think I've coded things in a modular enough way (hence putting almost everything, literally, in "modules") that I won't be running back to the starting line, and I have lots of reusable pieces that I really like as they are.
Monday, December 27, 2010
Some updates
Haven't updated in a while, so I thought I might write a post to give a little update on what's happening with Tengoku.
- If you've taken a look at the repository you might have noticed that a lot of change happened in the organization of the files; I ditched the colorfully named sub-libraries and cleaned out a lot of junk, including most of the bad, inaccurate docs lying around, old tests etc. The tests folder is very nice for doing experiments that double as tests.
- I finally settled on PascalCase for the entire thing and have been slowly updating the source code. I know I had just wrote that I decided that allcaps was the ultimate but then I read some code written by Pablo Reda implementing an IMGUI and found it to be very nice to read code with a mix of case, even in Forth, so decided that it was worth it. SwiftForth is case-insensitive so I haven't made the source perfectly consistent in regards to case but it definitely made it much more readable. I used to do all lower case mostly out of laziness, and still do when the code is throwaway. But hyphenation is an old Forth convention that I don't really like so I've started weeding that out - I still plan to use it though when I want to really distinguish the first word from the rest of the name of a function.
- New features are being added and changes are being made to the OOP framework all the time. I recently added two new flavors of classes to complement the Module keyword: Structure and Topic. These are cut-down versions of Module that don't support defining methods, to save memory, and to make intention clearer. Topic is meant to not have any properties or methods at all, only internal functions and static members. It's basically a convenient way of defining and working with a private wordlist that works with the same semantics as other classes.
- You can now open any class's internal words with the words [Using and Using], but I decided that I hated typing that so then I added [u and u] which is easier to type and u: which lets you do just one word, and then I recently checked up on Retro and saw its version, so I decided to experiment with the syntax "With: ... Without" . So there are three semantics for the same thing because I still haven't decided. Probably [u ... u] / u: in compile mode and With: ... Without in interpret mode.
- Inspection has been beefed up a lot and you can peek into all sorts of objects and get useful info. It's very nice to be able to quickly see what's going on in a group of variables - sort of a poor man's Visual Studio but nicer because you control exactly what the computer is showing you at any time. It's not hard, just put an object on the stack and add M?. They can be easily thrown into a dynamic textbox to have a live running update over-top the game. .S was enhanced at some point to by some black magic know what items on the stack are modules. Very nice and will be integrated into the HUD console I'm planning to add - that's a very old thing that I've had in every game engine experiment I've done since 2000 - but I found SwiftForth's sufficient up til this point. It will be very very cool, I've got lots of great ideas for it, such as a Fixed-point stack view and more like live-updated Current and Context state.
- I plan to enhance .Fields to show you summaries of non-classed fields based on their size ( if more than 4, judge from there - 8 would mean a double, more would mean an array ... of course it won't show the entire contents of a huge field, maybe just a size and the first and last bytes).
- I'm going to add a new feature that will work very nicely in Forth and will make certain pieces of OO code look like normal procedural code. It's called Property Sharing. You'll be able to define certain properties (I call them fields, but whatever) in a module as being Shared , and then in another module you'll be able to say that you want to share the properties of another module using a connection that you give a name to. Then you'll be able to invoke those fields without qualifying them at all, but they won't implicitly refer to This, instead they refer to the object that This (or the object on the stack, for external facing fields) is connected to. If you do qualify them, they will point to the properties not necessarily of that object but the one it is connected to. The connection is a plain old var that you must assign for the sharing to work. Of course, objects with shared fields automatically define their own connection to themselves. This will be used to streamline Entities and Components (currently the same thing is being accomplished with some ugly custom code), and make applications that utilize many interdependent objects.
- Now Tengoku has its own windowing and input code, called Iota. SDL 1.3 was too buggy and it wasn't being regularly updated so I moved to a library I just found out about called SFML. It's great, better than SDL, but it doesn't work on many computers with ATI cards. So I ported a small subset to Tengoku. Took me about 3 days, but, it works like a charm, and it can easily be swapped out for the real thing (when and if SFML matures to a level I feel that I could rely on), because I access it in the engine via a set of wrapper functions that match SFML. It's even compatible with SFML 1.6's data structures and events. Of course, the corresponding code was refactored heavily to make the task easier and came out to about ... 33% of the C++ code, I think. Iota was developed in a private repository that sat nested within the Tengoku folder structure in an ignored folder, so it could include some selected files that it needed, then it was migrated into and debugged using the various test programs.
- The GUI framework, called Studio, is fleshing out nicely. It is very easy to work with and you can do some fancy things like set up a game to support the mouse being able to pick up things and move them wherever you like.
I'm also doing some other things that I'm not sharing open-source. I'm developing the game, of course, but I don't want to talk about it. And I'm developing a set of tools built on the Studio (GUI) framework to work on graphics and maps. There's a comment in Studio.f that I forgot to take out that mentions ideas for this.
Guess that's about enough for today. I'm just glad the system is stable and mature enough now that I feel I can do anything with it.
- If you've taken a look at the repository you might have noticed that a lot of change happened in the organization of the files; I ditched the colorfully named sub-libraries and cleaned out a lot of junk, including most of the bad, inaccurate docs lying around, old tests etc. The tests folder is very nice for doing experiments that double as tests.
- I finally settled on PascalCase for the entire thing and have been slowly updating the source code. I know I had just wrote that I decided that allcaps was the ultimate but then I read some code written by Pablo Reda implementing an IMGUI and found it to be very nice to read code with a mix of case, even in Forth, so decided that it was worth it. SwiftForth is case-insensitive so I haven't made the source perfectly consistent in regards to case but it definitely made it much more readable. I used to do all lower case mostly out of laziness, and still do when the code is throwaway. But hyphenation is an old Forth convention that I don't really like so I've started weeding that out - I still plan to use it though when I want to really distinguish the first word from the rest of the name of a function.
- New features are being added and changes are being made to the OOP framework all the time. I recently added two new flavors of classes to complement the Module keyword: Structure and Topic. These are cut-down versions of Module that don't support defining methods, to save memory, and to make intention clearer. Topic is meant to not have any properties or methods at all, only internal functions and static members. It's basically a convenient way of defining and working with a private wordlist that works with the same semantics as other classes.
- You can now open any class's internal words with the words [Using and Using], but I decided that I hated typing that so then I added [u and u] which is easier to type and u: which lets you do just one word, and then I recently checked up on Retro and saw its version, so I decided to experiment with the syntax "With: ... Without" . So there are three semantics for the same thing because I still haven't decided. Probably [u ... u] / u: in compile mode and With: ... Without in interpret mode.
- Inspection has been beefed up a lot and you can peek into all sorts of objects and get useful info. It's very nice to be able to quickly see what's going on in a group of variables - sort of a poor man's Visual Studio but nicer because you control exactly what the computer is showing you at any time. It's not hard, just put an object on the stack and add M?. They can be easily thrown into a dynamic textbox to have a live running update over-top the game. .S was enhanced at some point to by some black magic know what items on the stack are modules. Very nice and will be integrated into the HUD console I'm planning to add - that's a very old thing that I've had in every game engine experiment I've done since 2000 - but I found SwiftForth's sufficient up til this point. It will be very very cool, I've got lots of great ideas for it, such as a Fixed-point stack view and more like live-updated Current and Context state.
- I plan to enhance .Fields to show you summaries of non-classed fields based on their size ( if more than 4, judge from there - 8 would mean a double, more would mean an array ... of course it won't show the entire contents of a huge field, maybe just a size and the first and last bytes).
- I'm going to add a new feature that will work very nicely in Forth and will make certain pieces of OO code look like normal procedural code. It's called Property Sharing. You'll be able to define certain properties (I call them fields, but whatever) in a module as being Shared , and then in another module you'll be able to say that you want to share the properties of another module using a connection that you give a name to. Then you'll be able to invoke those fields without qualifying them at all, but they won't implicitly refer to This, instead they refer to the object that This (or the object on the stack, for external facing fields) is connected to. If you do qualify them, they will point to the properties not necessarily of that object but the one it is connected to. The connection is a plain old var that you must assign for the sharing to work. Of course, objects with shared fields automatically define their own connection to themselves. This will be used to streamline Entities and Components (currently the same thing is being accomplished with some ugly custom code), and make applications that utilize many interdependent objects.
- Now Tengoku has its own windowing and input code, called Iota. SDL 1.3 was too buggy and it wasn't being regularly updated so I moved to a library I just found out about called SFML. It's great, better than SDL, but it doesn't work on many computers with ATI cards. So I ported a small subset to Tengoku. Took me about 3 days, but, it works like a charm, and it can easily be swapped out for the real thing (when and if SFML matures to a level I feel that I could rely on), because I access it in the engine via a set of wrapper functions that match SFML. It's even compatible with SFML 1.6's data structures and events. Of course, the corresponding code was refactored heavily to make the task easier and came out to about ... 33% of the C++ code, I think. Iota was developed in a private repository that sat nested within the Tengoku folder structure in an ignored folder, so it could include some selected files that it needed, then it was migrated into and debugged using the various test programs.
- The GUI framework, called Studio, is fleshing out nicely. It is very easy to work with and you can do some fancy things like set up a game to support the mouse being able to pick up things and move them wherever you like.
I'm also doing some other things that I'm not sharing open-source. I'm developing the game, of course, but I don't want to talk about it. And I'm developing a set of tools built on the Studio (GUI) framework to work on graphics and maps. There's a comment in Studio.f that I forgot to take out that mentions ideas for this.
Guess that's about enough for today. I'm just glad the system is stable and mature enough now that I feel I can do anything with it.
Monday, December 6, 2010
Took my own advice and removed an inappropriate use of polymorphism. There's a somewhat technical word EDATA which points other words to the ENTITY object for any object that implements the IENTITY interface. (Components implement it.) It was originally a method. Methods are slow. But SO many words called EDATA, crucial stuff like position. I changed EDATA to a code word; now all it does is fetch the address pointed to by THIS; the rule is, all IENTITY's have to store a pointer to an ENTITY in the first cell. Bind by memory layout instead of method table, get 35% speed gain.
Only Forth lets you break out of structure to make better use of resources so elegantly and painlessly. I think it's a preview into a future of less waste.
Only Forth lets you break out of structure to make better use of resources so elegantly and painlessly. I think it's a preview into a future of less waste.
Argh
OK, no, I'm not in good mood.
Turns out, spending all these months adding modern features to Forth has ended up making it harder to debug (and therefore write), more complicated, AND, slower. Well, maybe. It might be my fault for ab/using these new features. Not using them appropriately. I had a feeling about it, but I was so concerned about looking "legit" to the public, making the code all "high level" with lots of generics. I am not sure if anyone cares right now - which doesn't bother me at all, because it means I don't have to be "legit" to keep anyone's attention. Since no one is bothering me, and Plan A didn't lead to a big bang, I can take it wherever I want.
Turns out generics have a price. Turns out they complicate things if you don't use them appropriately; maybe one central polymorphism system is hugely unnecessary some of the time. Maybe I need to say OK here are places where it can make things easier (high level) and here is where it would actually be easier if I ignored them (low level).
Forth is assembly on steroids. It makes low level seem high level, makes it fit in the palm of your hand, makes it friendly; manageable. Forth likes dumb data types, and keeping things flat - not building systems upon systems, but instead making specialized systems out of bits and pieces - that's flatter design that doesn't necessarily hinder the complexity of your ideas but keeps them easier to test and change. If those low level bits and pieces are reliable and rich, starting "from scratch" should be less painful.
Admitedly, some systems are complex. But when you admit that, the fact that complex systems are hard to implement becomes a great motivation to simplify; not necessarily to not do your idea, but to let Forth help you do it in a dumb, flat, manageable way. If you're smart and confident about Forth, you'll get that working and then implement the complex stuff under the hood, invisible and irrelevent to the calling code.
And you shouldn't proliferate redundant datatypes (integers, fixed point, halfword, bytes), unless you really REALLY need to for some reason. Polymorphism helps manage that (and Forth hates lots of types since it has no typing system), but it has to be appropriate. For instance I know that it's prohibitive for building dynamic vertex lists, because each call to write out a vertex would be twice as expensive. But for strings, it makes sense because it's an inherently non-realtime thing that you do in short bursts. Polymorphism would help solve the problem of having to manipulate strings for external API's that love them, or for storing XML, or writing out Forth code :)
Forth has wordlists, a concept that lines up neatly with modular program design. I'm using them, but maybe not as well as I could be. Because sometimes I use polymorphism when reaching into a wordlist would be fine, and more efficient (direct call instead of a method call).
What's so weird is when everyone else says that making a program fast means "hard to read" or "complex", the old collision grid code was soooo much faster yet it was about the same size it is now. Less if you count the assembly code I had to write to make it reasonable.
I get hung up on loading everything at once, maybe it's a laziness; as if to say "you never know when something will break" so you compile everything in case something does. That is distracting; the human brain wasn't designed to work on more than one thing at once, because when it does, when I'm able to concentrate on one thing at a time, I get a lot more efficient. If I used wordlists more often I'd be forced to narrow focus because it's inconvenient to be constantly modifying the search order.
OK, so, what have we learned ... it's good to break it down, figure out what you've been doing in the broad scope, and what you're doing right now, and make the bits and pieces ("primitives") support that by being fast and easy to use. Perhaps wrap the polymorphic system AROUND primitives (same name, separate wordlist(s)) to virtualize them for higher level systems; but only when it's appropriate; probably great for modular plugin things.
Turns out, spending all these months adding modern features to Forth has ended up making it harder to debug (and therefore write), more complicated, AND, slower. Well, maybe. It might be my fault for ab/using these new features. Not using them appropriately. I had a feeling about it, but I was so concerned about looking "legit" to the public, making the code all "high level" with lots of generics. I am not sure if anyone cares right now - which doesn't bother me at all, because it means I don't have to be "legit" to keep anyone's attention. Since no one is bothering me, and Plan A didn't lead to a big bang, I can take it wherever I want.
Turns out generics have a price. Turns out they complicate things if you don't use them appropriately; maybe one central polymorphism system is hugely unnecessary some of the time. Maybe I need to say OK here are places where it can make things easier (high level) and here is where it would actually be easier if I ignored them (low level).
Forth is assembly on steroids. It makes low level seem high level, makes it fit in the palm of your hand, makes it friendly; manageable. Forth likes dumb data types, and keeping things flat - not building systems upon systems, but instead making specialized systems out of bits and pieces - that's flatter design that doesn't necessarily hinder the complexity of your ideas but keeps them easier to test and change. If those low level bits and pieces are reliable and rich, starting "from scratch" should be less painful.
Admitedly, some systems are complex. But when you admit that, the fact that complex systems are hard to implement becomes a great motivation to simplify; not necessarily to not do your idea, but to let Forth help you do it in a dumb, flat, manageable way. If you're smart and confident about Forth, you'll get that working and then implement the complex stuff under the hood, invisible and irrelevent to the calling code.
And you shouldn't proliferate redundant datatypes (integers, fixed point, halfword, bytes), unless you really REALLY need to for some reason. Polymorphism helps manage that (and Forth hates lots of types since it has no typing system), but it has to be appropriate. For instance I know that it's prohibitive for building dynamic vertex lists, because each call to write out a vertex would be twice as expensive. But for strings, it makes sense because it's an inherently non-realtime thing that you do in short bursts. Polymorphism would help solve the problem of having to manipulate strings for external API's that love them, or for storing XML, or writing out Forth code :)
Forth has wordlists, a concept that lines up neatly with modular program design. I'm using them, but maybe not as well as I could be. Because sometimes I use polymorphism when reaching into a wordlist would be fine, and more efficient (direct call instead of a method call).
What's so weird is when everyone else says that making a program fast means "hard to read" or "complex", the old collision grid code was soooo much faster yet it was about the same size it is now. Less if you count the assembly code I had to write to make it reasonable.
I get hung up on loading everything at once, maybe it's a laziness; as if to say "you never know when something will break" so you compile everything in case something does. That is distracting; the human brain wasn't designed to work on more than one thing at once, because when it does, when I'm able to concentrate on one thing at a time, I get a lot more efficient. If I used wordlists more often I'd be forced to narrow focus because it's inconvenient to be constantly modifying the search order.
OK, so, what have we learned ... it's good to break it down, figure out what you've been doing in the broad scope, and what you're doing right now, and make the bits and pieces ("primitives") support that by being fast and easy to use. Perhaps wrap the polymorphic system AROUND primitives (same name, separate wordlist(s)) to virtualize them for higher level systems; but only when it's appropriate; probably great for modular plugin things.
Optimizing: Global Collision System
Over the weekend, after working on the Studio application a bit, I got a little obsessed over the global collision system. Just a little bit of background; this is the system for partitioning the game's space into a grid of sectors and so culling down the number of objects it needs to test for overlap, using simple hitboxes. Consider it the broad collision phase. Also, this code was written for an engine from a few years ago, when I didn't use formal object orientation, or polymorphism, or components. As a result it was very fast - on a 1.7ghz machine, it could do 5000x5000 object tests at 60fps (and drawing ALL 5000 of them using Allegro, with CPU to spare).
After a first run of adaptation for Tengoku, the thing's performance was dismal; it couldn't do 5000x5000 at all, on a 3.3 ghz computer. (The old version also couldn't do objects bigger than 256x256, but the recent fixing of that is totally not the reason for the slowdown.) So, I felt forced to start optimizing to get it usable ... and here's what I did:
After a first run of adaptation for Tengoku, the thing's performance was dismal; it couldn't do 5000x5000 at all, on a 3.3 ghz computer. (The old version also couldn't do objects bigger than 256x256, but the recent fixing of that is totally not the reason for the slowdown.) So, I felt forced to start optimizing to get it usable ... and here's what I did:
- Sped up LIST:ADDTO; all it really needed to do was store the given thing into the last cell and increment the length field.
- Sped up LIST:EACH ; for some reason using an older, longer version of the routine sped things up just itterating objects by around 40%.
- Extended ENTITY (which affects memory usage globally ... so I guess that now I'm specializing for 2D...)
- Cached absolute rectangles for each object.
- Added BOX@ and used that instead of @BOX. (a dumbly named word) which leads to an assumption that the HITRECT attributes point to BOXes (which kind of defeats the purpose of IRECTANGLE ...)
- Sped up WORDMAP:LOOKUP by turning it into a CODE word; 2 hours. :(
All that work and eventually, performance was significantly improved. 1500x1500 objects in 7ms, when at first it couldn't do that many at all (it led to a spiral of death). It's still not good enough, in my opinion. I keep thinking something is wrong. I might do an more intense investigation into exactly what is happening; how many objects each object actually checks, how long each check takes...
But anyway, for the last thing I wanted to write about, I learned something amazing, and somewhat troubling. That the following piece of code turns a loop that took 1.5 ms into 6.5 ms!!!!
: ASDF DROP ;
: FDSA ['] ASDF MYLIST EACH ;
: FDSA ['] ASDF MYLIST EACH ;
Why? The only conclusion I can take from this is that modern, prefetching, superscalar processors hate, detest, and abhore very short subroutines. Granted most of my functions aren't that short but some get pretty close! It worries me because sometimes it has a good reason, such as the virtualizing mechanism that lets any component be treated as it's owner, where you have routines like this:
: POS EDATA >_POS ;
Which comes out to a CALL and an ADD instruction. Seems to me that this MUST MUST MUST be inlined. I'll try that out when I get home.
Friday, December 3, 2010
Wednesday, November 10, 2010
Speedups
Recently I optimized several parts of the SDK, mostly to be able to support a "bullet hell" type game (one of the ideas I'm kicking around.)
Here is a little summary of what I accomplished:
Here is a little summary of what I accomplished:
- Entities are much faster
- The heap is now faster and more memory efficient and lets you allocate up to
256MB16MB (up from 1MB!) - The Layer component was eliminated and folded into Entity so parenting is more efficient.
- Spritebatches were implemented; performance is not too shabby with 5000 moving sprites a frame on a 3.2ghz desktop. It's definitely CPU bottlenecked; eventually I'll offload sprite rendering completely to the GPU via....
- The Shader module; shaders will bring huge performance gains without a doubt :P
Tuesday, November 9, 2010
Providing Tengoku as a DLL
Today I considered that to make Tengoku accessible from any language, practically all you need is the system compiled as a dynamic library, and this API of 4 whopping functions. I think.
- ten_evaluate ( string ) : string
- ten_push ( int ) : void
- ten_pop () : int
- ten_wrap ( function ) : int
And maybe double and triple variants for ten_push and ten_pop.
Ten_wrap would take a native function and make it something that Tengoku can execute, since it makes extensive use of callbacks.
Ten_evaluate would take a simple string containing Forth "script", likely usually just a single function name, and return a string of whatever was output to the console. Since it would be against SwiftForth's commercial license, all code compilation words would be inaccessible; and you could only access game engine words. (So, all logic would be in your native language, which would be the point of it anyway.)
On second thought, that probably isn't it. You also need multitasking and messages, which right now are intrinsic to Plexi and Walnut, respectively. And a more efficient way of getting the address of objects and then their properties, else it would just be too slow to be more than a demo.
Maybe there can be a way of automatically go through and generate the DLL and headers using the stack comments.
Saturday, October 23, 2010
Starting to document things
http://code.google.com/p/tengoku-engine/wiki/Plexi
http://code.google.com/p/tengoku-engine/wiki/Jump
It's really slapdash, and I am not documenting every little thing. Just the stuff I need to have in my head or check.
http://code.google.com/p/tengoku-engine/wiki/Jump
It's really slapdash, and I am not documenting every little thing. Just the stuff I need to have in my head or check.
Wednesday, September 29, 2010
Update - abolishing ->
In the midst of a cleanup. Decided that I don't have a use for a revamped module internals wordset yet, instead I added a couple needed features.
1) Now you can jump to a method implementation in the source using the convention ":".
2) A new kind of method, the "xmethod", or "external method", methods intended to face outside, complementing their non-method counterparts. Work like normal methods but always requires the module to be on the stack. I'm in the process of abolishing "->", so this is one step towards that goal. The rule is, if a method is generally meant to be an interface to the OUTSIDE, make it external. If it is used internally only, make it a normal one. Once -> is gone though, { } clauses or AS will be the only ways to use normal methods.
3) Because you occasionally have to call external methods from a { } clause, I added a shorthand for THIS, the dollar sign ("$").
1) Now you can jump to a method implementation in the source using the convention "
2) A new kind of method, the "xmethod", or "external method", methods intended to face outside, complementing their non-method counterparts. Work like normal methods but always requires the module to be on the stack. I'm in the process of abolishing "->", so this is one step towards that goal. The rule is, if a method is generally meant to be an interface to the OUTSIDE, make it external. If it is used internally only, make it a normal one. Once -> is gone though, { } clauses or AS will be the only ways to use normal methods.
3) Because you occasionally have to call external methods from a { } clause, I added a shorthand for THIS, the dollar sign ("$").
Tuesday, September 28, 2010
1.41.3.0 Build
What's new in this one is mainly the global sprite collision system.
Posted on the Downloads page.
Posted on the Downloads page.
Monday, September 27, 2010
Tidbits
I recently added the last feature of global bounding box collisions, making all the so-far planned features complete.
The interesting bit is this; I found that Plexi's module-related punctuation ( { }, -> ) made programming too un-Forth-like, and created encumbrances in the system due to the one-size-fits-all nature. My solution; let the user customize the system for every problem. This is the way Forth does it from the ground up; I think that by keeping the module data structures as the "lingua franca" but allowing easy access to its internals will open up novel ways of using them that goes beyond what traditional OO systems let you do.
Sunday, September 26, 2010
Possibly transforming refactoring looming
I plan on taking the features of the module system, and "exploding" them. That's a term I just came up with. The best analogy is probably like toppling the tower of Babel. Everything you need still exists, it's just going to be even MORE horizontal. Actually in this case it's more like demolishing a 2-story house and making it more like a rancher. Or a big basketball court. OK I'm done with analogy-making tonight.
Subscribe to:
Posts (Atom)