tag:blogger.com,1999:blog-1994130783874175266.post7418721968779895789..comments2024-03-29T12:50:41.386+01:00Comments on bitsquid: development blog: A new data storage modelNiklashttp://www.blogger.com/profile/10055379994557504977noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-1994130783874175266.post-54083960759659103372011-07-31T22:51:08.398+02:002011-07-31T22:51:08.398+02:00@Mihai Our solution to that is to always export to...@Mihai Our solution to that is to always export to an intermediate format, not the final binary format.<br /><br />The intermediate format (JSON) is human readable and stable to version changes (since it is property-based it is quite easy to make it backwards _and_ forward compatible). The intermediate format is what is checked in to source control, so it can't be out-of-date.<br /><br />The JSON->binary conversion is done by our data compiler. The data compiler is built and distributed together with our game executable. (In fact it is a special version of that executable.) This means that when designers update they will always get a data compiler and a matching executable that agree on the binary format, so the binary format can never get out-of-sync.<br /><br />A change of the binary format triggers a recompile of the intermediate files (JSON->binary).Niklashttps://www.blogger.com/profile/10055379994557504977noreply@blogger.comtag:blogger.com,1999:blog-1994130783874175266.post-32154436548986821952011-07-17T20:45:10.159+02:002011-07-17T20:45:10.159+02:00We something like this ...the editor would export ...We something like this ...the editor would export the data in binary format and it would also export c structs so our game could load the data.<br />the problem was that designers would often update only the exe or only the data and then everything would be out of sync ....and it would crash :(Mihai Sebeahttps://www.blogger.com/profile/13663373445883513856noreply@blogger.comtag:blogger.com,1999:blog-1994130783874175266.post-69394972194231807092010-08-20T10:09:25.792+02:002010-08-20T10:09:25.792+02:00Srekel: I'm still toying with this idea and it...Srekel: I'm still toying with this idea and it hasn't been implemented in the engine. There we still use JSON files in the file system.<br /><br />Also, this is intended for production data, not for runtime data. The runtime data would be effecient binary blobs "compiled" from this data. This data would only be touched by editors and the data compiler.<br /><br />I would not generate classes/structs from this data. Classes would be defined in the editors and these classes would then save/load their data from the database.<br /><br />In pseudo code something like (sorry for the bad formatting):<br /><br />function GuiText:Load(db, id)<br />____self.id = id<br />____self.x = db.get(id, "x")<br />____self.y = db.get(id, "y")<br />____self.text = db.get(id, "text")<br />end<br /><br />The editors would be written in a language with decent reflection support (e.g. C#), which means that this could be completely automated.<br /><br />Objects in a set do not need to be of the same type. The database has no concept of "type".<br /><br />There would be both a generic tool and custom tools for examining/modifying the data. Using the generic tool would be similar to looking at XML-files or JSON-files in a text editor -- useful for debugging but not a very good way of manipulating the data. The custom tools would be your game editors and exporters -- level editor, animation editor, particle editor, gui editor, etc.<br /><br />As said above, the runtime engine will not touch this data, by the time it reaches the engine it will have been compiled to efficient binary blobs. There won't be any keys at all, just binary data matching the engine's structs that can be read directly into memory.Niklashttps://www.blogger.com/profile/10055379994557504977noreply@blogger.comtag:blogger.com,1999:blog-1994130783874175266.post-90056533904397643172010-08-20T09:47:36.426+02:002010-08-20T09:47:36.426+02:00amecky: Yes this is a NoSQL database, but that doe...amecky: Yes this is a NoSQL database, but that doesn't really say much, since NoSQL is such a vague term (encompassing anything that isn't a traditional relational database).<br /><br />To me, the simplifications and restrictions are what makes this scheme interesting. First, because a simple system is easy to understand, explain and reason about, which is very valuable in itself. Second, because these restrictions makes it possible to do conflict-free diffs and merges of database versions which means it can be used by multiple users in "offline" mode or as a revision control system.<br /><br />You could use a NoSQL database as a backend for implementing this scheme though. Tokyo Cabinet could be a good fit.Niklashttps://www.blogger.com/profile/10055379994557504977noreply@blogger.comtag:blogger.com,1999:blog-1994130783874175266.post-7824773644268293002010-08-20T08:54:06.755+02:002010-08-20T08:54:06.755+02:00How do you handle the data in the engine? Presumab...How do you handle the data in the engine? Presumably you want type safefy and efficient storage, so I'm guessing you'll generate classes/structs? <br /><br />I'm presuming that a set of objects must be of the same type?<br /><br />Have you considered how you'll be editing the data? Just a single tool with automatically generated tables, like a database tool, or custom tools for each type of data?<br /><br />I'm assuming that just because you manage the keys as strings in the tools chain, in the actual engine you'll either use hashed strings to access some sort of hashtable, or preferably typed pointers.Srekelhttps://www.blogger.com/profile/07324938919606033926noreply@blogger.comtag:blogger.com,1999:blog-1994130783874175266.post-70108340596125719172010-08-19T08:17:39.080+02:002010-08-19T08:17:39.080+02:00It seems that are talking about NoSQL here. Take a...It seems that are talking about NoSQL here. Take a look here http://nosql-database.org/ and you will find your concept has already been implemented. Well yours is a simplification of a nosql database. But it is basically the same. Maybe you will find some valuable ideas there as well.ameckyhttps://www.blogger.com/profile/10680169423101402419noreply@blogger.com