Building the plugin on OS X (was Re: The Include Macro)
Nicholas Pasternack
nicholas.pasternack at verizon.net
Mon Apr 16 23:48:54 UTC 2012
Hello Dave,
The new makefile does not work; I tried it on the current release of the plugin and the version of the plugin I made last time. I figured out how to fix the arch command on the macports version of gcc. The new makefile gives the following errors during the test_module_with_type and test_simple_compilation phase:
“
Stderr:
Undefined symbols for architecture i386:
"_PyType_Ready", referenced from:
_initmodule_with_type in ccGOhJyC.o
"_PyString_FromString", referenced from:
_example_Example_repr in ccGOhJyC.o
"_Py_InitModule4", referenced from:
_initmodule_with_type in ccGOhJyC.o
"_PyType_GenericNew", referenced from:
_example_ExampleType in ccGOhJyC.o
"_PyModule_AddObject", referenced from:
_initmodule_with_type in ccGOhJyC.o
ld: symbol(s) not found for architecture i386
collect2: ld returned 1 exit status
Undefined symbols for architecture x86_64:
"_PyType_Ready", referenced from:
_initmodule_with_type in ccL9484b.o
"_Py_InitModule4_64", referenced from:
_initmodule_with_type in ccL9484b.o
"_PyString_FromString", referenced from:
_example_Example_repr in ccL9484b.o
"_PyType_GenericNew", referenced from:
_example_ExampleType in ccL9484b.o
"_PyModule_AddObject", referenced from:
_initmodule_with_type in ccL9484b.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
lipo: can't open input file: /var/folders/3r/3rkFWJnFFkiA4+SfbiFLqU+++TI/-Tmp-//ccDcPtfa.out (No such file or directory)
“
The old makefile still works, and now it will be a fat binary (works with 32 and 64 bit machines). The directory error has been fixed now.
Nick
On Apr 9, 2012, at 8:09 PM, David Malcolm wrote:
> On Wed, 2012-03-28 at 16:25 -0400, Nicholas Pasternack wrote:
>> Hello Dave,
>
> Sorry; this got held up in the moderation queue due to the size of the
> attachments. My bad.
>
>> I'm attaching the git diff and Makefile to this email.
> The Makefile has changed considerably since the version your diff is
> against. In theory it ought to be possible to build by setting CC,
> PYTHON and PYTHON_CONFIG in the environment to suitable values;
> something like:
>
> $ make clean
> $ export CC=gcc-mp-4.6
> $ export PYTHON=python2.7
> $ export PYTHON_CONFIG=python2.7-config
> $ make
>
>> I looked into the issues with the Mac file system errors. As far as I
>> know, the only solutions are:
>> * building your own FS
>> * having two commits, one with a file that does not have that
>> kind of error
>> * changing the name of the code_d directory
>
> I've renamed the directories in git as of
> 6990bfc9d5aa6c425e65abea7f257839cc021641, so hopefully this works now.
>
>
>> On Mar 27, 2012, at 12:27 PM, David Malcolm wrote:
>>
>>> On Sun, 2012-03-25 at 14:58 -0400, Nicholas Pasternack wrote:
>>>
>>> [changing the Subject to "Building the plugin on OS X"]
>>>
>>>> Dave, thanks for the advice.
>>>>
>>>> That file was very helpful; I got the plugin to build on my
>>>> machine.
>>> Excellent.
>>>
>>> Can you send the current "git diff" of your working copy to this
>>> list?
>>>
>>> I just merged a set of Makefile fixes from Kevin Pyle:
>>> https://github.com/davidmalcolm/gcc-python-plugin/pull/2
>>> which probably clash horribly with the changes you've got locally.
>>> However, they ought to give us a better starting point from which to
>>> fix
>>> OS X compatibility.
>>>
>>> In particular, the compiler is now defined as the variable
>>> "CC" (rather
>>> than "GCC"), so if you set:
>>> $ CC=gcc-mp-4.6 make
>>> or:
>>> $ export CC=gcc-mp-4.6
>>> make
>>> you ought to be able to override things that way without needing to
>>> hack
>>> that up within the Makefile.
>>>
>>> As for the other changes, I'm not sure how to manually merge them
>>> without seeing your current "git diff".
>>>
>>>
>>>> Unfortunately, there are some problems; the plugin works on the
>>>> examples: show-passes, show-callgraph, and show-docs only.
>>>> Show-ssa,
>>>> show-gimple, and the libcpychecker all give seg faults. These seg
>>>> faults occur regardless of the c code being compiled (I tried
>>>> several
>>>> different c programs including one that just prints hello world).
>>> The patch I attached two posts ago is a nasty hack that got things
>>> to
>>> build (during an informal coding session at a conference), and I
>>> commented out some important things.
>>>
>>> IIRC, we ran into problems with fmemopen not being available, and
>>> crudely hacked these out.
>>>
>>> The plugin uses GCC's own prettyprinter hooks to print string
>>> representations of things, which it uses to implement str() and
>>> repr()
>>> hooks for the majority of the Python classes; the GCC code expects a
>>> FILE* to write the data to: we use fmemopen to capture the data
>>> internally in a buffer. IIRC I hacked this out, so you may be
>>> running
>>> into some stray NULL pointers for a FILE*, crashing things.
>>>
>>> But this is just a hunch: I'd have to see your diff to be sure.
>>>
>>> I've added some detailed notes on how to debug the plugin using gdb
>>> here:
>>> http://gcc-python-plugin.readthedocs.org/en/latest/getting-involved.html#debugging-the-plugin-s-c-code
>>> However, those cover my experiences on Linux (Fedora, specifically);
>>> you
>>> may run into extra issues I haven't thought of (sorry, I last owned
>>> an
>>> OS X machine back in the 10.0 days!).
>>>
>>>
>>>> For gcc-with python, when I compile the same c code without the
>>>> plugin (using macports gcc 4.6), I get no errors or warnings.
>>>> When I
>>>> built python.so, there were no errors or warnings either.
>>>> However,
>>>> when the object files were made, there were a lot of warnings.
>>>> Additionally, I'm not sure if my usage of libcpychecker was right:
>>>> I
>>>> used: ./gcc-with-cpychecker input.c script.py (I took some example
>>>> code and put it into the main plugin directory).
>>> The correct way to invoke the checker is simply:
>>> ./gcc-with-cpychecker input.c
>>> or:
>>> ./gcc-with-cpychecker REGULAR GCC ARGS GO HERE input.c
>>>
>>> There's also a separate script: gcc-with-python, which *does* take a
>>> script.py
>>> ./gcc-with-python script.py input.c
>>> or:
>>> ./gcc-with-python script.py REGULAR GCC ARGS GO HERE input.c
>>>
>>>> Here are some other things I had to change to build the plugin in
>>>> addition to the changes listed on the file you sent me:
>>>> * Needs python 2.7 (uses argparse etc.)
>>> Ooops, yes - thanks.
>>>
>>>> * gcc-with-python needed to be changed to: gcc-mp-4.6
>>>> -fplugin=python.so -fplugin-arg-python-script=$@
>>>> * gcc-python-docs needed to be changed to: ./gcc-with-python
>>>> examples/show-docs.py test.c
>>> I've now changed these to pick up the CC variable in the
>>> environment, if
>>> it exists, in commits 72dbbbd7183201952add5182579d318f6b689409 and
>>> 8b55a9f845ef9d78bf43d423682ae82c11fd5eeb.
>>>
>>>
>>>> * You need to have fmemopen.o pre-made: the header and c
>>>> source
>>>> I used is available here:
>>>> http://code.google.com/p/python-tesseract/source/browse/wiki/python-tesseract-0.7/?r=112
>>>> * In the Makefile, the command to build the plugin needs to
>>>> include fmemopen.o (I got a linking error when I didn't)
>>>> * The Makefile should have fmemopen.c in the source files
>>>> * Also in the Makefile: Python.framework/Versions/2.7/Python
>>>> does not exist; it should be changed
>>>> to: /Library/Frameworks/Python.framework/Versions/2.7/Python.
>>>> I also needed to leave out the arch commands to build it.
>>>> The full command to build the plugin (python.so) should
>>>> look
>>>> like:
>>>> * gcc-mp-4.6
>>>> -I/opt/local/lib/gcc46/gcc/x86_64-apple-darwin10/4.6.3/plugin/include -fPIC -fno-strict-aliasing -O2 -Wall -g -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -g -O2 -DNDEBUG -g -O3 -L/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -ldl -framework CoreFoundation -lpython2.7 -u _PyMac_Error /Library/Frameworks/Python.framework/Versions/2.7/Python -undefined dynamic_lookup -shared gcc-python.o gcc-python-attribute.o gcc-python-callbacks.o gcc-python-callgraph.o gcc-python-cfg.o gcc-python-closure.o gcc-python-diagnostics.o gcc-python-function.o gcc-python-gimple.o gcc-python-location.o gcc-python-option.o gcc-python-parameter.o gcc-python-pass.o gcc-python-pretty-printer.o gcc-python-rtl.o gcc-python-tree.o gcc-python-variable.o gcc-python-version.o gcc-python-wrapper.o autogenerated-callgraph.o autogenerated-cfg.o autogenerated-option.o autogenerated-function.o autogenerated-gimple.o autogenerated-location.o autogenerated-parameter.o autogenerated-pass.o autogenerated-pretty-printer.o autogenerated-rtl.o autogenerated-tree.o autogenerated-variable.o util-fmemopen.o fmemopen.o -o python.so
>>> How much of the above is emitted by suitable invocations of the
>>> python-config command?
>>>
>>>> * I needed to have all the .def files in the directory with
>>>> the
>>>> plugin (this may just be something wrong with the macports
>>>> gcc
>>>> 4.6).
>>> No idea, sorry.
>>>
>>>> If you want a full copy of the Makefile I used to build the
>>>> plugin,
>>>> please let me know.
>>> Yes please, but a "git diff" is likely to be more readable.
>>>
>>>> When I run the Makefile, it does not build the test-suite because
>>>> of
>>>> the seg faults. I think the problem is in the c source for the
>>>> plugin because both libcpychecker and gcc-with-python give seg
>>>> faults
>>>> (maybe fmemopen problems). I will continue working on the seg
>>>> fault
>>>> problem.
>>>> Do you have any ideas about how to fix the seg faults?
>>> (see notes above).
>>>
>>>
>>>> At some point, would you want me to make a Mac OSX install package
>>>> for
>>>> the plugin?
>>> Given that you're building the plugin against the MacPorts build of
>>> GCC,
>>> it seems to be most appropriate for the plugin itself to be added to
>>> MacPorts, rather than as a regular OS X install package.
>>>
>>> This is not my area of expertise, but information on this appears to
>>> be
>>> here:
>>> http://guide.macports.org/#development.creating-portfile
>>>
>>> Hope this all makes sense
>>> Dave
>>>
>>>>
>>>>
>>>> Nick
>>>>
>>>>
>>>> On Mar 21, 2012, at 9:22 PM, David Malcolm wrote:
>>>>
>>>>> On Wed, 2012-03-21 at 17:01 -0400, Nicholas Pasternack wrote:
>>>>>
>>>>> Hi Nicholas
>>>>>
>>>>> I've added comments inline below throughout.
>>>>>
>>>>>> The following header does not work on all UNIX machines or
>>>>>> machines
>>>>>> running multiple versions of python: #include <Python.h>
>>>>>>
>>>>>> Depending on what version of Python you use, the header needs
>>>>>> to
>>>>>> change to: #include <python(version).(release)/Python.h> For
>>>>>> instance, if you have python 2.7, the include line should
>>>>>> read:
>>>>>> #include <python2.7/Python.h>
>>>>>
>>>>> The line:
>>>>> #include <Python.h>
>>>>> is the standard idiom for including the C Python extension API.
>>>>> Locating the correct path for the header files (with the correct
>>>>> version) is done using compiler options.
>>>>>
>>>>> Normally this is done for gcc by passing a -I argument to the
>>>>> compiler.
>>>>> For example, -I/usr/include/python2.7
>>>>>
>>>>> You can determine the correct build flags by invoking
>>>>> python-config --cflags
>>>>>
>>>>> On my machine, this emits:
>>>>> -I/usr/include/python2.7 -I/usr/include/python2.7
>>>>> -fno-strict-aliasing
>>>>> -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
>>>>> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
>>>>> -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall
>>>>> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
>>>>> --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE
>>>>> -fPIC
>>>>> -fwrapv
>>>>>
>>>>>
>>>>> The Makefile tries to do this to extract these arguments, and it
>>>>> uses
>>>>> the variable PYTHON_CONFIG so that this can be overridden if you
>>>>> have
>>>>> multiple pythons installed. There may be bugs, of course.
>>>>>
>>>>>> To the best of my knowledge, there are no Macros from the gcc
>>>>>> that
>>>>>> define the Python version the user is running. However, I
>>>>>> know
>>>>>> the
>>>>>> following will be compatible with Mac computers:
>>>>>>
>>>>>> #ifdef __APPLE__
>>>>>> #include <python2.6/Python.h>
>>>>>> #else
>>>>>> #include <Python.h>
>>>>>> #endif
>>>>>
>>>>> Are you trying to get the code to run on an OS X box? Some of
>>>>> us
>>>>> spent
>>>>> some time at PyCon trying to get the code to build on such a
>>>>> machine.
>>>>>
>>>>> I'm attaching the (very hacky) patch that eventually got the
>>>>> thing
>>>>> to
>>>>> build, using the MacPorts build of gcc 4.6 (thanks to Daniel
>>>>> Lepage).
>>>>>
>>>>> It would be good to get this cleaned up and merged.
>>>>>
>>>>> Areas of pain we ran into:
>>>>> * the plugin's code uses "fmemopen" in a few places, which
>>>>> doesn't
>>>>> exist on OS X (we took some code from SCF as a quick and dirty
>>>>> way
>>>>> to
>>>>> get the code to run [1]; we've approached that project for
>>>>> permission to
>>>>> copy and relicense, but it's probably fairly easy to
>>>>> reimplement)
>>>>> * IIRC the:
>>>>> __typeof__ (decl_as_string) decl_as_string __attribute__
>>>>> ((weak))
>>>>> stuff that we do didn't work, so we simply disabled it for now
>>>>> * the change to
>>>>> tests/cpychecker/Py_BuildValue/code_d/correct/input.c
>>>>> is an OS X artifact: the filesystem is case-preserving but not
>>>>> case-sensitive, so code_d/* and code_D/* cannot coexist. On git
>>>>> checkout, one of them overwrites the other (and this seriously
>>>>> confuses
>>>>> git)
>>>>>
>>>>>
>>>>>> I think there should be a short explanatory note in the README
>>>>>> describing how to change the include line; if the user has
>>>>>> multiple
>>>>>> versions of Python, the code will not compile unless you
>>>>>> specifically
>>>>>> state what version of Python to use in the include line.
>>>>> The README.rst doesn't cover this, but docs/basics.rst does.
>>>>>
>>>>> Hope this is helpful
>>>>> Dave
>>>>>
>>>>> [1]
>>>>> <https://redmine.openinfosecfoundation.org/attachments/105/0001-fmemopen-wrapper-added-fix-compilation-problems-on-m.patch>
>>>>>>
>>>>>
>>>>> <os-x.diff>_______________________________________________
>>>>> gcc-python-plugin mailing list
>>>>> gcc-python-plugin at lists.fedorahosted.org
>>>>> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> gcc-python-plugin mailing list
>>>> gcc-python-plugin at lists.fedorahosted.org
>>>> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
>>>
>>>
>>> _______________________________________________
>>> gcc-python-plugin mailing list
>>> gcc-python-plugin at lists.fedorahosted.org
>>> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
>>>
>>
>> _______________________________________________
>> gcc-python-plugin mailing list
>> gcc-python-plugin at lists.fedorahosted.org
>> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
>
>
> _______________________________________________
> gcc-python-plugin mailing list
> gcc-python-plugin at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
More information about the gcc-python-plugin
mailing list