Building the plugin on OS X (was Re: The Include Macro)

David Malcolm dmalcolm at redhat.com
Tue Apr 10 00:09:37 UTC 2012


On Wed, 2012-03-28 at 16:25 -0400, Nicholas Pasternack wrote:
> Hello Dave,

Sorry; this got held up in the moderation queue due to the size of the
attachments.  My bad.

> I'm attaching the git diff and Makefile to this email.  
The Makefile has changed considerably since the version your diff is
against.  In theory it ought to be possible to build by setting CC,
PYTHON and PYTHON_CONFIG in the environment to suitable values;
something like:

  $ make clean
  $ export CC=gcc-mp-4.6
  $ export PYTHON=python2.7
  $ export PYTHON_CONFIG=python2.7-config
  $ make

> I looked into the issues with the Mac file system errors.  As far as I
> know, the only solutions are: 
>       * building your own FS 
>       * having two commits, one with a file that does not have that
>         kind of error
>       * changing the name of the code_d directory

I've renamed the directories in git as of
6990bfc9d5aa6c425e65abea7f257839cc021641, so hopefully this works now.


> On Mar 27, 2012, at 12:27 PM, David Malcolm wrote:
> 
> > On Sun, 2012-03-25 at 14:58 -0400, Nicholas Pasternack wrote:
> > 
> > [changing the Subject to "Building the plugin on OS X"]
> > 
> > > Dave, thanks for the advice.
> > > 
> > > That file was very helpful; I got the plugin to build on my
> > > machine.
> > Excellent.
> > 
> > Can you send the current "git diff" of your working copy to this
> > list?
> > 
> > I just merged a set of Makefile fixes from Kevin Pyle:
> >  https://github.com/davidmalcolm/gcc-python-plugin/pull/2
> > which probably clash horribly with the changes you've got locally.
> > However, they ought to give us a better starting point from which to
> > fix
> > OS X compatibility.
> > 
> > In particular, the compiler is now defined as the variable
> > "CC" (rather
> > than "GCC"), so if you set:
> >  $ CC=gcc-mp-4.6 make 
> > or:
> >  $ export CC=gcc-mp-4.6
> >  make
> > you ought to be able to override things that way without needing to
> > hack
> > that up within the Makefile.
> > 
> > As for the other changes, I'm not sure how to manually merge them
> > without seeing your current "git diff".
> > 
> > 
> > >  Unfortunately,  there are some problems; the plugin works on the
> > > examples: show-passes, show-callgraph, and show-docs only.
> > >  Show-ssa,
> > > show-gimple, and the libcpychecker all give seg faults.  These seg
> > > faults occur regardless of the c code being compiled (I tried
> > > several
> > > different c programs including one that just prints hello world).
> > The patch I attached two posts ago is a nasty hack that got things
> > to
> > build (during an informal coding session at a conference), and I
> > commented out some important things.
> > 
> > IIRC, we ran into problems with fmemopen not being available, and
> > crudely hacked these out.  
> > 
> > The plugin uses GCC's own prettyprinter hooks to print string
> > representations of things, which it uses to implement str() and
> > repr()
> > hooks for the majority of the Python classes; the GCC code expects a
> > FILE* to write the data to: we use fmemopen to capture the data
> > internally in a buffer.  IIRC I hacked this out, so you may be
> > running
> > into some stray NULL pointers for a FILE*, crashing things.
> > 
> > But this is just a hunch: I'd have to see your diff to be sure.
> > 
> > I've added some detailed notes on how to debug the plugin using gdb
> > here:
> > http://gcc-python-plugin.readthedocs.org/en/latest/getting-involved.html#debugging-the-plugin-s-c-code
> > However, those cover my experiences on Linux (Fedora, specifically);
> > you
> > may run into extra issues I haven't thought of (sorry, I last owned
> > an
> > OS X machine back in the 10.0 days!).
> > 
> > 
> > >  For gcc-with python, when I compile the same c code without the
> > > plugin (using macports gcc 4.6), I get no errors or warnings.
> > >  When I
> > > built python.so, there were no errors or warnings either.
> > >  However,
> > > when the object files were made, there were a lot of warnings.
> > > Additionally, I'm not sure if my usage of libcpychecker was right:
> > > I
> > > used: ./gcc-with-cpychecker input.c script.py (I took some example
> > > code and put it into the main plugin directory).  
> > The correct way to invoke the checker is simply:
> >  ./gcc-with-cpychecker input.c
> > or:
> >  ./gcc-with-cpychecker REGULAR GCC ARGS GO HERE input.c
> > 
> > There's also a separate script: gcc-with-python, which *does* take a
> > script.py
> >  ./gcc-with-python script.py input.c
> > or:
> >  ./gcc-with-python script.py REGULAR GCC ARGS GO HERE input.c
> > 
> > > Here are some other things I had to change to build the plugin in
> > > addition to the changes listed on the file you sent me:
> > >      * Needs python 2.7 (uses argparse etc.)
> > Ooops, yes - thanks.
> > 
> > >      * gcc-with-python needed to be changed to: gcc-mp-4.6
> > >        -fplugin=python.so -fplugin-arg-python-script=$@
> > >      * gcc-python-docs needed to be changed to: ./gcc-with-python
> > >        examples/show-docs.py test.c
> > I've now changed these to pick up the CC variable in the
> > environment, if
> > it exists, in commits 72dbbbd7183201952add5182579d318f6b689409 and
> > 8b55a9f845ef9d78bf43d423682ae82c11fd5eeb.
> > 
> > 
> > >      * You need to have fmemopen.o pre-made: the header and c
> > > source
> > >        I used is available here:
> > >        http://code.google.com/p/python-tesseract/source/browse/wiki/python-tesseract-0.7/?r=112
> > >      * In the Makefile, the command to build the plugin needs to
> > >        include fmemopen.o (I got a linking error when I didn't)
> > >      * The Makefile should have fmemopen.c in the source files
> > >      * Also in the Makefile: Python.framework/Versions/2.7/Python
> > >        does not exist; it should be changed
> > >        to: /Library/Frameworks/Python.framework/Versions/2.7/Python.
> > >         I also needed to leave out the arch commands to build it.
> > >         The full command to build the plugin (python.so) should
> > > look
> > >        like: 
> > >              * gcc-mp-4.6
> > >                -I/opt/local/lib/gcc46/gcc/x86_64-apple-darwin10/4.6.3/plugin/include -fPIC -fno-strict-aliasing -O2 -Wall -g -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -g -O2 -DNDEBUG -g -O3  -L/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -ldl -framework CoreFoundation -lpython2.7 -u _PyMac_Error /Library/Frameworks/Python.framework/Versions/2.7/Python -undefined dynamic_lookup -shared gcc-python.o gcc-python-attribute.o gcc-python-callbacks.o gcc-python-callgraph.o gcc-python-cfg.o gcc-python-closure.o gcc-python-diagnostics.o gcc-python-function.o gcc-python-gimple.o gcc-python-location.o gcc-python-option.o gcc-python-parameter.o gcc-python-pass.o gcc-python-pretty-printer.o gcc-python-rtl.o gcc-python-tree.o gcc-python-variable.o gcc-python-version.o gcc-python-wrapper.o autogenerated-callgraph.o autogenerated-cfg.o autogenerated-option.o autogenerated-function.o autogenerated-gimple.o autogenerated-location.o autogenerated-parameter.o autogenerated-pass.o autogenerated-pretty-printer.o autogenerated-rtl.o autogenerated-tree.o autogenerated-variable.o util-fmemopen.o fmemopen.o -o python.so
> > How much of the above is emitted by suitable invocations of the
> > python-config command?
> > 
> > >      * I needed to have all the .def files in the directory with
> > > the
> > >        plugin (this may just be something wrong with the macports
> > > gcc
> > >        4.6).
> > No idea, sorry.
> > 
> > > If you want a full copy of the Makefile I used to build the
> > > plugin,
> > > please let me know.  
> > Yes please, but a "git diff" is likely to be more readable.
> > 
> > > When I run the Makefile, it does not build the test-suite because
> > > of
> > > the seg faults.    I think the problem is in the c source for the
> > > plugin because both libcpychecker and gcc-with-python give seg
> > > faults
> > > (maybe fmemopen problems).   I will continue working on the seg
> > > fault
> > > problem.  
> > > Do you have any ideas about how to fix the seg faults?  
> > (see notes above).
> > 
> > 
> > > At some point, would you want me to make a Mac OSX install package
> > > for
> > > the plugin?
> > Given that you're building the plugin against the MacPorts build of
> > GCC,
> > it seems to be most appropriate for the plugin itself to be added to
> > MacPorts, rather than as a regular OS X install package.
> > 
> > This is not my area of expertise, but information on this appears to
> > be
> > here:
> > http://guide.macports.org/#development.creating-portfile
> > 
> > Hope this all makes sense
> > Dave
> > 
> > > 
> > > 
> > > Nick
> > > 
> > > 
> > > On Mar 21, 2012, at 9:22 PM, David Malcolm wrote:
> > > 
> > > > On Wed, 2012-03-21 at 17:01 -0400, Nicholas Pasternack wrote:
> > > > 
> > > > Hi Nicholas
> > > > 
> > > > I've added comments inline below throughout.
> > > > 
> > > > > The following header does not work on all UNIX machines or
> > > > > machines
> > > > > running multiple versions of python: #include <Python.h> 
> > > > > 
> > > > > Depending on what version of Python you use, the header needs
> > > > > to
> > > > > change to: #include <python(version).(release)/Python.h>  For
> > > > > instance, if you have python 2.7, the include line should
> > > > > read:
> > > > > #include <python2.7/Python.h>
> > > > 
> > > > The line:
> > > > #include <Python.h>
> > > > is the standard idiom for including the C Python extension API.
> > > > Locating the correct path for the header files (with the correct
> > > > version) is done using compiler options.
> > > > 
> > > > Normally this is done for gcc by passing a -I argument to the
> > > > compiler.
> > > > For example, -I/usr/include/python2.7
> > > > 
> > > > You can determine the correct build flags by invoking
> > > > python-config --cflags
> > > > 
> > > > On my machine, this emits:
> > > > -I/usr/include/python2.7 -I/usr/include/python2.7
> > > > -fno-strict-aliasing
> > > > -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
> > > > -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
> > > > -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall
> > > > -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
> > > > --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE
> > > > -fPIC
> > > > -fwrapv
> > > > 
> > > > 
> > > > The Makefile tries to do this to extract these arguments, and it
> > > > uses
> > > > the variable PYTHON_CONFIG so that this can be overridden if you
> > > > have
> > > > multiple pythons installed.  There may be bugs, of course.
> > > > 
> > > > > To the best of my knowledge, there are no Macros from the gcc
> > > > > that
> > > > > define the Python version the user is running.  However, I
> > > > > know
> > > > > the
> > > > > following will be compatible with Mac computers:
> > > > > 
> > > > > #ifdef __APPLE__
> > > > > #include <python2.6/Python.h>
> > > > > #else
> > > > > #include <Python.h>
> > > > > #endif
> > > > 
> > > > Are you trying to get the code to run on an OS X box?  Some of
> > > > us
> > > > spent
> > > > some time at PyCon trying to get the code to build on such a
> > > > machine.
> > > > 
> > > > I'm attaching the (very hacky) patch that eventually got the
> > > > thing
> > > > to
> > > > build, using the MacPorts build of gcc 4.6 (thanks to Daniel
> > > > Lepage).
> > > > 
> > > > It would be good to get this cleaned up and merged.
> > > > 
> > > > Areas of pain we ran into:
> > > > * the plugin's code uses "fmemopen" in a few places, which
> > > > doesn't
> > > > exist on OS X (we took some code from SCF as a quick and dirty
> > > > way
> > > > to
> > > > get the code to run [1]; we've approached that project for
> > > > permission to
> > > > copy and relicense, but it's probably fairly easy to
> > > > reimplement)
> > > > * IIRC the:
> > > >  __typeof__ (decl_as_string) decl_as_string __attribute__
> > > > ((weak))
> > > >  stuff that we do didn't work, so we simply disabled it for now
> > > > * the change to
> > > > tests/cpychecker/Py_BuildValue/code_d/correct/input.c
> > > > is an OS X artifact: the filesystem is case-preserving but not
> > > > case-sensitive, so code_d/* and code_D/* cannot coexist. On git
> > > > checkout, one of them overwrites the other (and this seriously
> > > > confuses
> > > > git)
> > > > 
> > > > 
> > > > > I think there should be a short explanatory note in the README
> > > > > describing how to change the include line; if the user has
> > > > > multiple
> > > > > versions of Python, the code will not compile unless you
> > > > > specifically
> > > > > state what version of Python to use in the include line.
> > > > The README.rst doesn't cover this, but docs/basics.rst does.
> > > > 
> > > > Hope this is helpful
> > > > Dave
> > > > 
> > > > [1]
> > > > <https://redmine.openinfosecfoundation.org/attachments/105/0001-fmemopen-wrapper-added-fix-compilation-problems-on-m.patch>
> > > > > 
> > > > 
> > > > <os-x.diff>_______________________________________________
> > > > gcc-python-plugin mailing list
> > > > gcc-python-plugin at lists.fedorahosted.org
> > > > https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
> > > > 
> > > 
> > > 
> > > _______________________________________________
> > > gcc-python-plugin mailing list
> > > gcc-python-plugin at lists.fedorahosted.org
> > > https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
> > 
> > 
> > _______________________________________________
> > gcc-python-plugin mailing list
> > gcc-python-plugin at lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/gcc-python-plugin
> > 
> 
> _______________________________________________
> gcc-python-plugin mailing list
> gcc-python-plugin at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/gcc-python-plugin




More information about the gcc-python-plugin mailing list