Building the plugin on OS X (was Re: The Include Macro)

Nicholas Pasternack nicholas.pasternack at verizon.net
Mon Apr 16 23:48:54 UTC 2012


Hello Dave,
 
The new makefile does not work; I tried it on the current release of the plugin and the version of the plugin I made last time.  I figured out how to fix the arch command on the macports version of gcc.  The new makefile gives the following errors during the test_module_with_type and test_simple_compilation phase: 
 
“
Stderr:
    Undefined symbols for architecture i386:
      "_PyType_Ready", referenced from:
          _initmodule_with_type in ccGOhJyC.o
      "_PyString_FromString", referenced from:
          _example_Example_repr in ccGOhJyC.o
      "_Py_InitModule4", referenced from:
          _initmodule_with_type in ccGOhJyC.o
      "_PyType_GenericNew", referenced from:
          _example_ExampleType in ccGOhJyC.o
      "_PyModule_AddObject", referenced from:
          _initmodule_with_type in ccGOhJyC.o
    ld: symbol(s) not found for architecture i386
    collect2: ld returned 1 exit status
    Undefined symbols for architecture x86_64:
      "_PyType_Ready", referenced from:
          _initmodule_with_type in ccL9484b.o
      "_Py_InitModule4_64", referenced from:
          _initmodule_with_type in ccL9484b.o
      "_PyString_FromString", referenced from:
          _example_Example_repr in ccL9484b.o
      "_PyType_GenericNew", referenced from:
          _example_ExampleType in ccL9484b.o
      "_PyModule_AddObject", referenced from:
          _initmodule_with_type in ccL9484b.o
    ld: symbol(s) not found for architecture x86_64
    collect2: ld returned 1 exit status
    lipo: can't open input file: /var/folders/3r/3rkFWJnFFkiA4+SfbiFLqU+++TI/-Tmp-//ccDcPtfa.out (No such file or directory)
“
 
The old makefile still works, and now it will be a fat binary (works with 32 and 64 bit machines).  The directory error has been fixed now.
 
Nick

On Apr 9, 2012, at 8:09 PM, David Malcolm wrote:

> On Wed, 2012-03-28 at 16:25 -0400, Nicholas Pasternack wrote:
>> Hello Dave,
> 
> Sorry; this got held up in the moderation queue due to the size of the
> attachments.  My bad.
> 
>> I'm attaching the git diff and Makefile to this email.  
> The Makefile has changed considerably since the version your diff is
> against.  In theory it ought to be possible to build by setting CC,
> PYTHON and PYTHON_CONFIG in the environment to suitable values;
> something like:
> 
>  $ make clean
>  $ export CC=gcc-mp-4.6
>  $ export PYTHON=python2.7
>  $ export PYTHON_CONFIG=python2.7-config
>  $ make
> 
>> I looked into the issues with the Mac file system errors.  As far as I
>> know, the only solutions are: 
>>      * building your own FS 
>>      * having two commits, one with a file that does not have that
>>        kind of error
>>      * changing the name of the code_d directory
> 
> I've renamed the directories in git as of
> 6990bfc9d5aa6c425e65abea7f257839cc021641, so hopefully this works now.
> 
> 
>> On Mar 27, 2012, at 12:27 PM, David Malcolm wrote:
>> 
>>> On Sun, 2012-03-25 at 14:58 -0400, Nicholas Pasternack wrote:
>>> 
>>> [changing the Subject to "Building the plugin on OS X"]
>>> 
>>>> Dave, thanks for the advice.
>>>> 
>>>> That file was very helpful; I got the plugin to build on my
>>>> machine.
>>> Excellent.
>>> 
>>> Can you send the current "git diff" of your working copy to this
>>> list?
>>> 
>>> I just merged a set of Makefile fixes from Kevin Pyle:
>>> https://github.com/davidmalcolm/gcc-python-plugin/pull/2
>>> which probably clash horribly with the changes you've got locally.
>>> However, they ought to give us a better starting point from which to
>>> fix
>>> OS X compatibility.
>>> 
>>> In particular, the compiler is now defined as the variable
>>> "CC" (rather
>>> than "GCC"), so if you set:
>>> $ CC=gcc-mp-4.6 make 
>>> or:
>>> $ export CC=gcc-mp-4.6
>>> make
>>> you ought to be able to override things that way without needing to
>>> hack
>>> that up within the Makefile.
>>> 
>>> As for the other changes, I'm not sure how to manually merge them
>>> without seeing your current "git diff".
>>> 
>>> 
>>>> Unfortunately,  there are some problems; the plugin works on the
>>>> examples: show-passes, show-callgraph, and show-docs only.
>>>> Show-ssa,
>>>> show-gimple, and the libcpychecker all give seg faults.  These seg
>>>> faults occur regardless of the c code being compiled (I tried
>>>> several
>>>> different c programs including one that just prints hello world).
>>> The patch I attached two posts ago is a nasty hack that got things
>>> to
>>> build (during an informal coding session at a conference), and I
>>> commented out some important things.
>>> 
>>> IIRC, we ran into problems with fmemopen not being available, and
>>> crudely hacked these out.  
>>> 
>>> The plugin uses GCC's own prettyprinter hooks to print string
>>> representations of things, which it uses to implement str() and
>>> repr()
>>> hooks for the majority of the Python classes; the GCC code expects a
>>> FILE* to write the data to: we use fmemopen to capture the data
>>> internally in a buffer.  IIRC I hacked this out, so you may be
>>> running
>>> into some stray NULL pointers for a FILE*, crashing things.
>>> 
>>> But this is just a hunch: I'd have to see your diff to be sure.
>>> 
>>> I've added some detailed notes on how to debug the plugin using gdb
>>> here:
>>> http://gcc-python-plugin.readthedocs.org/en/latest/getting-involved.html#debugging-the-plugin-s-c-code
>>> However, those cover my experiences on Linux (Fedora, specifically);
>>> you
>>> may run into extra issues I haven't thought of (sorry, I last owned
>>> an
>>> OS X machine back in the 10.0 days!).
>>> 
>>> 
>>>> For gcc-with python, when I compile the same c code without the
>>>> plugin (using macports gcc 4.6), I get no errors or warnings.
>>>> When I
>>>> built python.so, there were no errors or warnings either.
>>>> However,
>>>> when the object files were made, there were a lot of warnings.
>>>> Additionally, I'm not sure if my usage of libcpychecker was right:
>>>> I
>>>> used: ./gcc-with-cpychecker input.c script.py (I took some example
>>>> code and put it into the main plugin directory).  
>>> The correct way to invoke the checker is simply:
>>> ./gcc-with-cpychecker input.c
>>> or:
>>> ./gcc-with-cpychecker REGULAR GCC ARGS GO HERE input.c
>>> 
>>> There's also a separate script: gcc-with-python, which *does* take a
>>> script.py
>>> ./gcc-with-python script.py input.c
>>> or:
>>> ./gcc-with-python script.py REGULAR GCC ARGS GO HERE input.c
>>> 
>>>> Here are some other things I had to change to build the plugin in
>>>> addition to the changes listed on the file you sent me:
>>>>     * Needs python 2.7 (uses argparse etc.)
>>> Ooops, yes - thanks.
>>> 
>>>>     * gcc-with-python needed to be changed to: gcc-mp-4.6
>>>>       -fplugin=python.so -fplugin-arg-python-script=$@
>>>>     * gcc-python-docs needed to be changed to: ./gcc-with-python
>>>>       examples/show-docs.py test.c
>>> I've now changed these to pick up the CC variable in the
>>> environment, if
>>> it exists, in commits 72dbbbd7183201952add5182579d318f6b689409 and
>>> 8b55a9f845ef9d78bf43d423682ae82c11fd5eeb.
>>> 
>>> 
>>>>     * You need to have fmemopen.o pre-made: the header and c
>>>> source
>>>>       I used is available here:
>>>>       http://code.google.com/p/python-tesseract/source/browse/wiki/python-tesseract-0.7/?r=112
>>>>     * In the Makefile, the command to build the plugin needs to
>>>>       include fmemopen.o (I got a linking error when I didn't)
>>>>     * The Makefile should have fmemopen.c in the source files
>>>>     * Also in the Makefile: Python.framework/Versions/2.7/Python
>>>>       does not exist; it should be changed
>>>>       to: /Library/Frameworks/Python.framework/Versions/2.7/Python.
>>>>        I also needed to leave out the arch commands to build it.
>>>>        The full command to build the plugin (python.so) should
>>>> look
>>>>       like: 
>>>>             * gcc-mp-4.6
>>>>               -I/opt/local/lib/gcc46/gcc/x86_64-apple-darwin10/4.6.3/plugin/include -fPIC -fno-strict-aliasing -O2 -Wall -g -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -g -O2 -DNDEBUG -g -O3  -L/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -ldl -framework CoreFoundation -lpython2.7 -u _PyMac_Error /Library/Frameworks/Python.framework/Versions/2.7/Python -undefined dynamic_lookup -shared gcc-python.o gcc-python-attribute.o gcc-python-callbacks.o gcc-python-callgraph.o gcc-python-cfg.o gcc-python-closure.o gcc-python-diagnostics.o gcc-python-function.o gcc-python-gimple.o gcc-python-location.o gcc-python-option.o gcc-python-parameter.o gcc-python-pass.o gcc-python-pretty-printer.o gcc-python-rtl.o gcc-python-tree.o gcc-python-variable.o gcc-python-version.o gcc-python-wrapper.o autogenerated-callgraph.o autogenerated-cfg.o autogenerated-option.o autogenerated-function.o autogenerated-gimple.o autogenerated-location.o autogenerated-parameter.o autogenerated-pass.o autogenerated-pretty-printer.o autogenerated-rtl.o autogenerated-tree.o autogenerated-variable.o util-fmemopen.o fmemopen.o -o python.so
>>> How much of the above is emitted by suitable invocations of the
>>> python-config command?
>>> 
>>>>     * I needed to have all the .def files in the directory with
>>>> the
>>>>       plugin (this may just be something wrong with the macports
>>>> gcc
>>>>       4.6).
>>> No idea, sorry.
>>> 
>>>> If you want a full copy of the Makefile I used to build the
>>>> plugin,
>>>> please let me know.  
>>> Yes please, but a "git diff" is likely to be more readable.
>>> 
>>>> When I run the Makefile, it does not build the test-suite because
>>>> of
>>>> the seg faults.    I think the problem is in the c source for the
>>>> plugin because both libcpychecker and gcc-with-python give seg
>>>> faults
>>>> (maybe fmemopen problems).   I will continue working on the seg
>>>> fault
>>>> problem.  
>>>> Do you have any ideas about how to fix the seg faults?  
>>> (see notes above).
>>> 
>>> 
>>>> At some point, would you want me to make a Mac OSX install package
>>>> for
>>>> the plugin?
>>> Given that you're building the plugin against the MacPorts build of
>>> GCC,
>>> it seems to be most appropriate for the plugin itself to be added to
>>> MacPorts, rather than as a regular OS X install package.
>>> 
>>> This is not my area of expertise, but information on this appears to
>>> be
>>> here:
>>> http://guide.macports.org/#development.creating-portfile
>>> 
>>> Hope this all makes sense
>>> Dave
>>> 
>>>> 
>>>> 
>>>> Nick
>>>> 
>>>> 
>>>> On Mar 21, 2012, at 9:22 PM, David Malcolm wrote:
>>>> 
>>>>> On Wed, 2012-03-21 at 17:01 -0400, Nicholas Pasternack wrote:
>>>>> 
>>>>> Hi Nicholas
>>>>> 
>>>>> I've added comments inline below throughout.
>>>>> 
>>>>>> The following header does not work on all UNIX machines or
>>>>>> machines
>>>>>> running multiple versions of python: #include <Python.h> 
>>>>>> 
>>>>>> Depending on what version of Python you use, the header needs
>>>>>> to
>>>>>> change to: #include <python(version).(release)/Python.h>  For
>>>>>> instance, if you have python 2.7, the include line should
>>>>>> read:
>>>>>> #include <python2.7/Python.h>
>>>>> 
>>>>> The line:
>>>>> #include <Python.h>
>>>>> is the standard idiom for including the C Python extension API.
>>>>> Locating the correct path for the header files (with the correct
>>>>> version) is done using compiler options.
>>>>> 
>>>>> Normally this is done for gcc by passing a -I argument to the
>>>>> compiler.
>>>>> For example, -I/usr/include/python2.7
>>>>> 
>>>>> You can determine the correct build flags by invoking
>>>>> python-config --cflags
>>>>> 
>>>>> On my machine, this emits:
>>>>> -I/usr/include/python2.7 -I/usr/include/python2.7
>>>>> -fno-strict-aliasing
>>>>> -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
>>>>> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
>>>>> -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall
>>>>> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
>>>>> --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE
>>>>> -fPIC
>>>>> -fwrapv
>>>>> 
>>>>> 
>>>>> The Makefile tries to do this to extract these arguments, and it
>>>>> uses
>>>>> the variable PYTHON_CONFIG so that this can be overridden if you
>>>>> have
>>>>> multiple pythons installed.  There may be bugs, of course.
>>>>> 
>>>>>> To the best of my knowledge, there are no Macros from the gcc
>>>>>> that
>>>>>> define the Python version the user is running.  However, I
>>>>>> know
>>>>>> the
>>>>>> following will be compatible with Mac computers:
>>>>>> 
>>>>>> #ifdef __APPLE__
>>>>>> #include <python2.6/Python.h>
>>>>>> #else
>>>>>> #include <Python.h>
>>>>>> #endif
>>>>> 
>>>>> Are you trying to get the code to run on an OS X box?  Some of
>>>>> us
>>>>> spent
>>>>> some time at PyCon trying to get the code to build on such a
>>>>> machine.
>>>>> 
>>>>> I'm attaching the (very hacky) patch that eventually got the
>>>>> thing
>>>>> to
>>>>> build, using the MacPorts build of gcc 4.6 (thanks to Daniel
>>>>> Lepage).
>>>>> 
>>>>> It would be good to get this cleaned up and merged.
>>>>> 
>>>>> Areas of pain we ran into:
>>>>> * the plugin's code uses "fmemopen" in a few places, which
>>>>> doesn't
>>>>> exist on OS X (we took some code from SCF as a quick and dirty
>>>>> way
>>>>> to
>>>>> get the code to run [1]; we've approached that project for
>>>>> permission to
>>>>> copy and relicense, but it's probably fairly easy to
>>>>> reimplement)
>>>>> * IIRC the:
>>>>> __typeof__ (decl_as_string) decl_as_string __attribute__
>>>>> ((weak))
>>>>> stuff that we do didn't work, so we simply disabled it for now
>>>>> * the change to
>>>>> tests/cpychecker/Py_BuildValue/code_d/correct/input.c
>>>>> is an OS X artifact: the filesystem is case-preserving but not
>>>>> case-sensitive, so code_d/* and code_D/* cannot coexist. On git
>>>>> checkout, one of them overwrites the other (and this seriously
>>>>> confuses
>>>>> git)
>>>>> 
>>>>> 
>>>>>> I think there should be a short explanatory note in the README
>>>>>> describing how to change the include line; if the user has
>>>>>> multiple
>>>>>> versions of Python, the code will not compile unless you
>>>>>> specifically
>>>>>> state what version of Python to use in the include line.
>>>>> The README.rst doesn't cover this, but docs/basics.rst does.
>>>>> 
>>>>> Hope this is helpful
>>>>> Dave
>>>>> 
>>>>> [1]
>>>>> <https://redmine.openinfosecfoundation.org/attachments/105/0001-fmemopen-wrapper-added-fix-compilation-problems-on-m.patch>
>>>>>> 
>>>>> 
>>>>> <os-x.diff>_______________________________________________
>>>>> gcc-python-plugin mailing list
>>>>> gcc-python-plugin at lists.fedorahosted.org
>>>>> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> gcc-python-plugin mailing list
>>>> gcc-python-plugin at lists.fedorahosted.org
>>>> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
>>> 
>>> 
>>> _______________________________________________
>>> gcc-python-plugin mailing list
>>> gcc-python-plugin at lists.fedorahosted.org
>>> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
>>> 
>> 
>> _______________________________________________
>> gcc-python-plugin mailing list
>> gcc-python-plugin at lists.fedorahosted.org
>> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
> 
> 
> _______________________________________________
> gcc-python-plugin mailing list
> gcc-python-plugin at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin



More information about the gcc-python-plugin mailing list