Embending Python into C/C++ : how to send a C structure to Python ?

[ManuTOO]

Hello,

I just began to dig into Python, and although the language itself seems to be nice, and the bases of embedding into C/C++ easy enough (although not too easy), I have a super hard time to find a tutorial, a doc, or anything to let me know how to do this :
I want to send a pointer to a structure from my C code to a Python function, and then modify the C structure from within Python.

The idea is that I have my Player State + my Game State, and I want to send 2 pointers, 1 for each state to a Python function that will modify the Player State according to some scripted game logic...

All I could find about this matter was a few comments about people telling they spent days to try to achieve something like this, and finally they went to use LUA...

It seems SWIG could do the job, but in its doc, there's nothing about passing C structure as argument to Python functions; they only describe direct use of these structures from within Python.

I had a look at Boost.Python, which is mainly for extending Python through C++, with almost nothing to embed Python with C++...

I very wonder why something so basic & useful is never described in all docs & tutorials I came through till now...
So if anyone could give me an hand on this, I'd be very thankful...

PS: my favorites links so far to startup with Python :
Learn Python in 10 minutes : http://www.poromenos.org/tutorials/python? (actually, it's more like a little hour )
Embedding Python in Your C Programs : http://www.linuxjournal.com/article/8497 (quite good tuto to know the basis to embed Python within C/C++)
Python Extension writing : http://www.ragestorm.net/tutorials/25/pyext.html (gives the basis to add new type to Python, it looks like the way to go to add C structures as new types)

[ManuTOO]

Just while I were looking for a more "Python place" to post my previous query, I found out the comp.lang.python newsgroup :
http://groups.google.com/group/comp....ture+argument#
Once again, another programmer who was stuck on the same problem than me, who "just" needed 1 week to solve it...
It looks like I'm going to walk in his steps, and hopefully by following his solution, I'd need less time than him to produce something actually useful...

[lobotony]

using boost::python, can't you just put the pointer into a python var?
Something like:

struct GameState {};
GameState gameState;
boost::python::globals["gameState"] = &gameState;

You need to provide the python mapping for GameState of course.
That's at least how it works with luabind and what I gathered from a quick glance over the boost python docs.

hth,
Tony

[ManuTOO]

This :
"boost:: python::globals["gameState"] = &gameState;"
is not C++ ...
Maybe it's C# ..?

Anyway, I had more than a glance a Boost.Python, and I didn't see anything like this (I might have looked in the wrong place, though... :S )

And yup, 50% of the problem is to have my structure recognized from within Python, which seems to be done by "object = MyObjectPtr(o)" when using Swig... Going to check that today, fingers crossed...

PS: yup, it ain't easy to code with my fingers crossed

[jomanto]

This :
"boost:: python::globals["gameState"] = &gameState;"
is not C++ ...
Maybe it's C# ..?

This is actually perfectly running code from Tony's and my C++ sources...

After a quick look into boost:: python docs one can come up with a solution like this (you would pack this into modules and stuff for production code):

#include <iostream>
#include <string>
#include <boost/python.hpp>

using namespace boost::python;

struct World
{
  std::string msg;
  void greet()
  {
    std::cout << msg << std::endl;
  }
};

int main (int argc, char * const argv[])
{
  try
  {
    // Initialize context
    Py_Initialize();

    // Retrieve the main module.
    object main_module((handle<>(borrowed(PyImport_AddModule("__main__")))));

    // Retrieve the main module's namespace
    object main_namespace(main_module.attr("__dict__"));
    
    // Expose World class
    main_namespace["World"] = class_<World>("World")
                                .def_readwrite("msg", &World::msg)
                                .def("greet", &World::greet);

    // Expose World instance
    World world;
    main_namespace["world"] = ptr(&world);

    // Run script
    handle<> ignored(( PyRun_String("world.msg = 'Hello World!' \n"
                                    "world.greet()              \n",
                                    Py_file_input,
                                    main_namespace.ptr(),
                                    main_namespace.ptr() ) ));
    
    // Finalize context
    Py_Finalize();
  }
  catch( error_already_set )
  {
    PyErr_Print();
  }
  
  return 0;
}

Edit: I just saw that you want to pass the structure into python. For that you could do the following (replace from "// Expose World instance"):

    // Define python function
    handle<> ignored(( PyRun_String("def use_world(world):         \n"
                                    "   world.msg = 'Hello World!' \n"
                                    "   world.greet()              \n",
                                    Py_file_input,
                                    main_namespace.ptr(),
                                    main_namespace.ptr() ) ));
    
    // Create World instance
    World world;

    // Get python function...
    object use_world = main_namespace["use_world"];
    
    // ...and call it
    use_world(world);

cheers,
Timo

[ManuTOO]

Hello jomanto, your "World" is not exactly a structure, it's a class with everything in public, and if I recall well, it's an ease of the language to let structures behave like classes.
Me, I meant a struct, in the 1st C meaning, ie: a gathering a variables, and possibly other structures.

So I don't need to expose functions. I just want to expose the variables (which can be structures as well), and I don't want to do it manually (I already have plenty and if I add more, I don't want to do maintenance).

But thanks for ur example, as it'd be a good start, and better than nothing if I can't succeed with Swig...


PS: my bad for the "boost:: python::globals["gameState"] ", of course it can be done in C++ if "globals" is an object from a class with an overloaded [] operator.

[lobotony]

manutoo,

I don't believe that any slight technical differences between structs and classes are relevant in your case.
If you want to expose a nested hierarchy of objects into your scripting environment, you do just that. Prerequisite is wrapping all of the involved structs/classes, so that they are known to your interpreter. If you don't want to do that manually, your best bet is using a tool like swig or pyste (for boost::python).

[ManuTOO]

sure, struct variables & class members are totally equivalent in this case (and in most cases if not all, I guess), it's just when I saw ur example, I just saw the function at 1st, and needed long time to notice the msg...

And thanks to confirm my guess to go on with Swig to avoid some manual work...

[ManuTOO]

Ok, I finally did it, and as I expected, once it's done it looks easy to do.
What is not easy is to gather a huge tone on information split a bit everywhere, so I'm going to write this mini-tutorial to help anyone who will be in my case in the future.

GOAL:
====
You already have a full working game engine done in C/C++, and you want to extend it by using Python as a scripting language to modify some game states, entity states, etc..., by sending pointer of your structure states to Python.

I assume you're already a bit familiar with Python (version used = 2.5.2).
ie: at least, u know this => http://www.poromenos.org/tutorials/python?

I'm using Swig v1.3.36 to wrap my C stuff to be used in Python : http://www.swig.org/ .
It seems to be the most easy one to use, although the doc & the beginner guide is really repelling at 1st... :-S (especially, there's not a word about how it works when embedding Python; all examples are done by extending it)

You have to create an interface file, that Swig will parse to create a .c wrap file that you'll have to insert into your project.
For our case, we'll include this .c wrap file directly into our main code.

Here the interface file "MyProject.i" for Swig :

%module MyProject   // Name of your project

%inline %{       // the inline keyword allows to have only 1 declaration, both used by Swig, and then copied as it to the .c wrap file
extern int MyPrint(char *Str);   // to easily get some text from Python
extern int PythonError(PyObject *pObj);   // To catch Python Error ; although, MyPrint could be used instead, it'd work in the same way

struct SPythoned   // Our test structure
{
	float x, y, z;
	int Stat[32];
	char *pName, ShortName[32];
};

%}

You create the wrap .c file using this command :
Swig -python MyProject.i

You can add the file "MyProject.i" to your Visual Studio project, and do this custom step build :
[PathToSwig]\Swig.exe -python "$(InputPath)"
Output = $(InputName)_wrap.c

Now, you have MyProject_wrap.c .

Here the Python script file "PyTest.py" to test the function & structure ; in it we define the Python function TestObject that receives an object which is a pointer to an instance of our C structure SPythoned :

import MyProject

def TestObject(o):
	MyProject.MyPrint(o.pName + "\n")
	MyProject.MyPrint(o.ShortName)		# + String doesn't work with ShortName
	MyProject.MyPrint("\n")
	o.ShortName = "yaataa!!"
	o.x = 1
	o.y = o.x * 2.5
	o.z = o.x + o.y

But we will also need to redirect the Python error output, coz PyErr_Print() uses it, and there's no other way to get this output (it seems that Python developers "forgot" to add a PyErr_PrintToBuffer(char *Buffer) or something like this... :-S ).

Here the Python script "Error.Py" to do the redirection :

import MyProject, sys

class Redirect:
	def write(self, s):
		MyProject.PythonError(s)

def Redirection():
	sys.stderr = sys.stdout = Redirect()

And here the C++ code, to test all of this :

#include <stdio.h>
#include <Python.h>

extern "C"	// only needed when compiling in C++
{
	#include "MyProject_wrap.c"
}

// struct SPythoned is defined within MyProject_wrap.c just included above

extern "C" int MyPrint(char *Str)
{
	return printf(Str);
}

extern "C" int PythonError(PyObject *pObj)
{
	char *Str = PyString_AsString(pObj);
	return printf(Str);
}

void CheckPythonError(void)
{
	if (PyErr_Occurred() != NULL)
	{
		PyErr_Print();
		PyErr_Clear();
	}
}

void PythonTest(void)
{
	freopen("PyTest.Log", "w", stdout);	// Redirect output, so our printf() will go into "PyTest.Log"

	Py_Initialize();	// Init Python
	SWIG_init();		// Initialise SWIG types
	init_MyProject();		// Init our project wrapped by Swig

		//=== Redirect Python error output
	int Ok = PyRun_SimpleString("import Error\n"
										"Error.Redirection()\n");

		//=== Do a little test
	PyRun_SimpleString("import MyProject\n"
							"from time import time,ctime\n"
							"Str = 'Today is ' + ctime(time()) + '\\n'\n"
							"MyProject.MyPrint(Str)\n");
	CheckPythonError();

		//=== Test our structure
	SPythoned Object;
	PyObject *pMod, *pGlobalDict, *pFunc, *pResult, *pArg;

	Object.pName = "Toto";
	strcpy(Object.ShortName, "T.");

	pMod = PyImport_ImportModule("PyTest");	// Load "PyTest.py" (it will create a compiled version "PyTest.pyc")

	if (pMod)
	{
		pGlobalDict = PyModule_GetDict(pMod);	// Get main dictionary

		if (pGlobalDict)
		{
			pFunc = PyDict_GetItemString(pGlobalDict, "TestObject");	// Look for function TestObject

			if (pFunc)
			{
				pArg = SWIG_NewPointerObj((void*)&Object, SWIGTYPE_p_SPythoned, 1);	// Get Python Object for our Structure Pointer

				if (pArg)
				{
					pResult = PyObject_CallFunction(pFunc, "O", pArg);
					CheckPythonError();
					Py_CLEAR(pResult);

					printf("x = %g; y = %g; z = %g; Name = %s / %s\n",
												Object.x, Object.y, Object.z,
												Object.pName, Object.ShortName);
				}
			}
		}
	}

	Py_Finalize();
}

That's all !

You can do the same with C++ classes, and with structures within structures. Swig can parse & wrap most C/C++ declarations.

Note: To build your project in debug mode, you'll need python25_d.lib & python25_d.dll . To get these files, you must download the Python sources, and build the PCBuild in debug. For me, it failed using VS2005EE, but worked with VS2008.

I hope this mini-tutorial will help..!