Messages

[Templated Code generation]

This all started with trying to understand changes to server/src/entities_game_service/database_guild/outpost/plc.cpp/h


Templated Code generation
There exist xslt transforms for generating many of the interfaces through the server and common src trees.  The respective cpp/hfiles actually contain "DO NOT EDIT" notices.
I've been working through understanding what is going on there, specifically starting from merging changes that all seemed to start with database.xml.
There is one readme in the shard_unifier_service that kicks things off... it points to xslt_proc and generate_module_interface.xslt.  This interface transform matches code generated across many of the server folders and in the common game_share.. each with an xml descriptor that is very easily revived. 

Herewith are my notes if anyone wants to look at it, or is already familiar with it!

-----

=== Generated files: xslt ===
Somes files are generated from XML sources,
eg:
  server/shard_unifier_services' nel_database_mapping.cpp/h  and database_mapping.cpp/h
 
Editing these is not recommended (noted in the source!)  These files can be HUGE..
Updates to the structures and code is indicated to be done through making the necessary updates to the .xml files and running the xml through xsltproc (with an appropriate conversion) to generate the updated source.

Tool:
  sudo apt-get install xsltproc

Generation notes:
  The origional xslt transforms (eg: common/src/game_share/generate_module_interface.xslt) is aruably a little outdated.. (note use of auto_ptr)
    (updating while reviewing to ensure generated output at the very least matches repo source)

  Notes on Generators:
    common/src/game_share/generate_module_interface.xslt
      This transform does interface definitions for types.. wrapping enums with string getters and setters, member gets/sets and general object guff
      Source xmls fed to this transform are in various folders for sub components.
    - supports : header cpp php
    - updated this to use the CUniquePtr from deprecated auto_ptr as per repo code
    - added (int32) casts as per repo code
    - added nilPtr rename as per repo code
    - updates to correct this transform for c++11 compatiblility (*done - verfied new generation matches origional repo source!)
    Supported params:
        --stringparam output <header|cpp|php>
        --stringparam filename {target filename minux extension}


    common/src/game_share/generate_client_db.xslt
      This transform seems to be very specifically for the server and client databases.  Output is HUGE
      Source xmls fed to this transform are in various folders for sub components.
    - supports : cpp-server header-server cpp-client header-client
    Supported params:
        --stringparam output <header|cpp|php>
        --stringparam side <client|server>
        --stringparam filename {target filename minux extension}
        --stringparam bank <PLC|GUILD|OUTPOST>

  NB+++ General Results:
    The scripts could simplify code maintenance.. and simplify mods just based on updates to key files.
    It may be well worth organising the code generation with these transforms with some scripts to automate their use.


    === SERVER ===
 
   : server/src/shard_unifier_service
      Files that are identified to be generated:
        nel_database_mapping.cpp/h >> TF:generate_module_interface.xslt IN:nel_database_mapping.xml (*confirmed)
        database_mapping.cpp/h >> TF:generate_module_interface.xslt IN:database_mapping.xml (*confirmed)

      HowTo: Run within the server/src/shard_unifier_service folder...

        Generation source : nel_database_mapping.xml

Code Sélectionner Étendre


xsltproc -o nel_database_mapping.cpp --stringparam output cpp --stringparam filename nel_database_mapping ../../../common/src/game_share/generate_module_interface.xslt nel_database_mapping.xml
xsltproc -o nel_database_mapping.h --stringparam output header --stringparam filename nel_database_mapping ../../../common/src/game_share/generate_module_interface.xslt nel_database_mapping.xml

       
Generation source : database_mapping.xml

Code Sélectionner Étendre


xsltproc -o database_mapping.cpp --stringparam output cpp --stringparam filename database_mapping ../../../common/src/game_share/generate_module_interface.xslt database_mapping.xml
xsltproc -o database_mapping.h --stringparam output header --stringparam filename database_mapping ../../../common/src/game_share/generate_module_interface.xslt database_mapping.xml


   : server/src/entities_game_service
      Files that are identified to be generated:
        database_guild.cpp/h >> TF:generate_client_db.xslt IN:common/data_common/database.xml
        database_outpost.cpp/h >> TF:generate_client_db.xslt IN:common/data_common/database.xml
        database_plr.cpp/h >> TF:generate_client_db.xslt IN:common/data_common/database.xml

      Sources:
        database.xml >> common/data_common/database.xml
        -- used for : database_ [plr, guild, outpost] generation!
       
      HowTo: Run within the server/src/entities_game_service folder...
        Generation source : nel_database_mapping.xml

Code Sélectionner Étendre


xsltproc -o database_plr.cpp --stringparam output cpp --stringparam side server --stringparam filename database_plr --stringparam bank PLR ../../../common/src/game_share/generate_client_db.xslt ../../../common/data_common/database.xml
xsltproc -o database_plr.h --stringparam output header --stringparam side server --stringparam filename database_plr --stringparam bank PLR ../../../common/src/game_share/generate_client_db.xslt ../../../common/data_common/database.xml


Code Sélectionner Étendre


xsltproc -o database_guild.cpp --stringparam output cpp --stringparam side server --stringparam filename database_guild --stringparam bank GUILD ../../../common/src/game_share/generate_client_db.xslt ../../../common/data_common/database.xml
xsltproc -o database_guild.h --stringparam output header --stringparam side server --stringparam filename database_guild --stringparam bank GUILD ../../../common/src/game_share/generate_client_db.xslt ../../../common/data_common/database.xml


Code Sélectionner Étendre


xsltproc -o database_outpost.cpp --stringparam output cpp --stringparam side server --stringparam filename database_outpost --stringparam bank OUTPOST ../../../common/src/game_share/generate_client_db.xslt ../../../common/data_common/database.xml
xsltproc -o database_outpost.h --stringparam output header --stringparam side server --stringparam filename database_outpost --stringparam bank OUTPOST ../../../common/src/game_share/generate_client_db.xslt ../../../common/data_common/database.xml


          Note - client side does not seem to generate anything.
            There are no mode definitions for mode="header-client" or mode="cpp-client".
          eg:

Code Sélectionner Étendre


xsltproc -o database_plr_client_test.cpp --stringparam output cpp --stringparam side client --stringparam filename database_plr_client_test --stringparam bank PLR ../../../common/src/game_share/generate_client_db.xslt ../../../common/data_common/database.xml
xsltproc -o database_plr_client_test.h --stringparam output header --stringparam side client --stringparam filename database_plr_client_test --stringparam bank PLR ../../../common/src/game_share/generate_client_db.xslt ../../../common/data_common/database.xml


          Notes:
            Updates in database.xml from generated code that is in repo
              set: TARGET/CONTEXT_MENU/COUNTER=I5 (from I4 - match repo)
              add: EXCHANGE,RECEIVE,TEMP,SHARE,TRADING/
                          CREATE_TIME:I32 (to match repo)
                          SERIAL,I32 (to match repo)
              add: PACK_ANIMAL
              check: GUILD/FAME/count == 7 (old was 6.. see pvp_clan.h)
              set: PACKANIMAL/BEAST/count=7 (from 4 - match git repo)
              add: PACKANIMAL/BEAST/NAME:I32 (to match repo)
              check: FAME/PLAYER/count == 7 (old was 6.. see pvp_clan.h)
         
          NB+++Result:
            The output of running these is correct and complete updated code for
              database_plr.cpp/h database_guild.cpp/h database_outpost.cpp/h
            based on changes to the common/data_common/database.xml...
         
         
   !TODO!
   : server/src/admin_modules
      Files that are identified to be generated:
        admin_modules_itf.cpp/h

   !TODO!
   : server/src/patchman_service
      Files that are identified to be generated:
        module_admin_itf.cpp/h
        re_module_itf.cpp/h
        rr_module_itf.cpp/h
        spa_module_itf.cpp/h
        spm_module_itf.cpp/h
        spt_module_itf.cpp/h

   !TODO!
   : server/src/server_share
      Files that are identified to be generated:
        backup_service_itf.cpp/h
        char_name_mapper_itf.cpp/h
        chat_unifier_itf.cpp/h
        command_executor_itf.cpp/h
        entity_locator_itf.cpp/h
        entity_locator_itf.cpp/h
        logger_service_itf.cpp/h
        login_service_itf.cpp/h/php
        mail_forum_itf.cpp/h
       
       
    === COMMON ===

   : common/src/game_share
      Files that are identified to be generated:
        --- lua
        item_special_effect.cpp/h >> lua.. TF:enum_generator.lua IN:item_special_effect.lua (*eyeballed... this seems to be the only use case thus far for the lua convertor.. if more instances are found then perhaps an xslt transform should be on the cards)
        Note: quite a few of the enums follow the same structure.. whether this is spawn from the original use of the script or just devs rolling through as the enum type is simple enough to not really require a generator..
        --- xslt
        character_sync_itf.cpp/h >> TF:generate_module_interface.xslt IN:character_sync_itf.xml (*confirmed h/cpp, however manual repo mods change ucstring for Fullname to const ucstring&... need to verify concern)
        msg_ais_egs_gen.h >> TF:generate_module_interface.xslt IN:msg_ais_egs_gen.xml (*confirmed h, can gen cpp but is has no functional content.  Manual repo edits for new member: Organization get/set methods.  Fixed by updating xml source.)
        r2_modules_itf.cpp/h >> TF:generate_module_interface.xslt IN:r2_modules_itf.xml (*confirmed cpp/h - minor compile cleanup mods in repo src)
        r2_share_itf.cpp/h >> TF:generate_module_interface.xslt IN:r2_share_itf.xml (*confirmed cpp/h - minor compile cleanup mods in repo src)
        ring_session_manager_itf.cpp/h >> TF:generate_module_interface.xslt IN:ring_session_manager_itf.xml (*confirmed cpp/h)
        ring_session_manager_itf.php >> TF:(??) IN:(??)

      HowTo: Run within the common/src/game_share folder...
        Generation source : item_special_effect.lua
          (?? Lua script)

        Generation source : character_sync_itf.xml
          Note: fixed pathing

Code Sélectionner Étendre


xsltproc -o character_sync_itf.cpp --stringparam output cpp --stringparam filename character_sync_itf ./generate_module_interface.xslt character_sync_itf.xml
xsltproc -o character_sync_itf.h --stringparam output header --stringparam filename character_sync_itf ./generate_module_interface.xslt character_sync_itf.xml


        Generation source : msg_ais_egs_gen.xml
          Note: fixed pathing

Code Sélectionner Étendre


xsltproc -o msg_ais_egs_gen.h --stringparam output header --stringparam filename msg_ais_egs_gen ./generate_module_interface.xslt msg_ais_egs_gen.xml


        Generation source : r2_modules_itf.xml

Code Sélectionner Étendre


xsltproc -o r2_modules_itf.cpp --stringparam output cpp --stringparam filename r2_modules_itf ./generate_module_interface.xslt r2_modules_itf.xml
xsltproc -o r2_modules_itf.h --stringparam output header --stringparam filename r2_modules_itf ./generate_module_interface.xslt r2_modules_itf.xml


        Generation source : r2_share_itf.xml

Code Sélectionner Étendre

xsltproc -o r2_share_itf.cpp --stringparam output cpp --stringparam filename r2_share_itf ./generate_module_interface.xslt r2_share_itf.xml
xsltproc -o r2_share_itf.h --stringparam output header --stringparam filename r2_share_itf ./generate_module_interface.xslt r2_share_itf.xml


        Generation source : ring_session_manager_itf.xml
          Note: fixed pathing, shard count enum

Code Sélectionner Étendre


xsltproc -o ring_session_manager_itf.cpp --stringparam output cpp --stringparam filename ring_session_manager_itf ./generate_module_interface.xslt ring_session_manager_itf.xml
xsltproc -o ring_session_manager_itf.h --stringparam output header --stringparam filename ring_session_manager_itf ./generate_module_interface.xslt ring_session_manager_itf.xml


-------


So, after all that, it builds... lol, a little more automation and perhaps a little less work down the line for some mods

~/n/Sigma

Thanks for the welcome    I hope the language barrier is not an issue.. so far google translate has been good to help with viewing the content you have.  I hope that my posts could translate as well into French!

For the most part, much of the complexity I have hit has been with the varying versions and states of repositories and getting familiar with the general project structure.  It is apparent that there was ordered process behind the original build, I very much wish I could see an original setup lol.  There are Makefile references and a great hint in the tools/scripts/linux folder to the original chain.  So much of the work thus far has been trying to form a rational (and scripted) framework .. lol,   as you well said.. titanic task! 
I just got the full chain working and the client logging in to the character creation... after fighting the db and shard unifier over the the TPublicLevel enum naming ul_* vs pl_* in the db enums.. and lastly the ring session entry in the db.  (Again - your site notes were the key  )  The next hurdle was the faction count with the addition of the marauder..  There is just such a tremendous amount to grasp and follow, let alone trace and resolve.  Thankfully the db is quite simple and the code followable.

I've been accumulating a lot of notes and fixes for the rather terrifying construction that is the code base I've been merging and assembling along the way while wrestling with the compiler and deps at times.  I do hope to rationalise the effort into hopefully useful contributions.

I really want to get to understand the concepts of the shard/domain/session... playing with the db session I get the sense that this is effectively the instancing engine.. but what I don't understand yet is how the db configuration eg: shard/domain IDs tie back to the assets and on disk configuration.  Browsing through the files you can see remnants of the original per race starter island configs and holes where I assume the mainland was stripped .. and at other layers where presumably other shard configs would reside.. but what is not apparent yet is the layering for all of this.  It IS there.. I'm sure, lol.  Need to find the key.   

~/nSigma/

Hi All

Just have to give an absolutely massive thanks for this project and the time and effort to make a rational and coherent build chain with resources for the Ryzom core..  I have been working on following the docs as a learning exercise to familiarize myself with the ryzom core for fun and given what is out there the best description I can give is that it is fractured, especially on the server side.
Over the weekend I ended up pulling and comparing three whole repos from various stages of official to piece together a viable build... lol. 
Your docs and git repositories have been awesome.. truly.  The most compete working reference I have had.

I started with trying to replicate the old server VM from scratch using Debian bullseye x64 as a base.. this to run a similarly build client.. and my gods has this been a rabbit hole of cosmic proportions! .. I finally got the server runnable  yesterday.. (not completely mind, but all the binary, config and system pieces in place -  even revived and updated the old Nel server Admin web in the process, ported from the VM I had completely missed all references to it in the repositories ... not knowing where to look .. sigh)   I was working on the AMS today while tracing the client login  when I found your site and repositories today and you have the resources piece of the puzzle I have been pondering as well.. just fantastic.   .  Up to there I was basically code diving to try understand what on earth was going on to stitch together an understanding of the layers..  and there are a LOT.  Lol.  Learning through immersion is far more educational... but a functional reference is something really useful to have to avoid beating your head against the proverbial wall.

I am looking forward to getting through this and honestly, finding your work has given me a good kick of confidence that it is possible, as well as details on some of the workings and resources. 
I am really looking forward to working through more of the content you guys have pieced together and hope (one day soon to understand and have a complete chain and setup from scratch)... then to the next fun part!

~/nSigma/