Using Python.jl and PyCall.jl together

[PallHaraldsson]

If I do:

using Python
using PyCall

the latter segfaults (at least on 1.6-rc1). For the reverse order I do get an error.

I don't personally need both to work, just curious if they could (probably not). That's not really the end-goal, but some packages do have PyCall as a dependency, e.g. PyPlot.jl and Pandas.jl. Is there a good way to support such, as it seems this package has a similar API as PyCall?

[cjdoris]

I just changed Python so it works even if libpython was already initialised, e.g. by PyCall. Provided they are using the same libpython I don't see why there should be any problem.

I get no segfault (windows, miniconda, Julia 1.5). What's your segfault?

[PallHaraldsson]

These options can take a lot of time off startup (from 31 sec for me), but I get same segfault either way:

$ ~/julia-1.6.0-rc1/bin/julia --startup-file=no -O0 --compile=min -q
julia> @time using Python
  7.174756 seconds (19.01 M allocations: 1.108 GiB, 8.90% gc time)

julia> @time using PyCall
ERROR: InitError: PyError (PyImport_ImportModule) <class 'SystemError'>
SystemError('initialization of _opcode did not return an extension module',)

signal (11): Segmentation fault
in expression starting at none:0
PyObject_Call at /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0 (unknown line)
unknown function (ip: 0x7fc83d0f3efe)
Allocations: 22058024 (Pool: 22050568; Big: 7456); GC: 28
Segmentation fault (core dumped)

If I start with PyCall, I get this error for your package:

julia> using Python
ERROR: InitError: Python: <error while printing type: Python.PyException(Python.PyRef(Ptr{Python.CPython.PyObject} @0x00007f38c279e3a0), Python.PyRef(Ptr{Python.CPython.PyObject} @0x00007f38cc0b4450), Python.PyRef(Ptr{Python.CPython.PyObject} @0x00007f38cc0b6800))>: <error while printing value: Python.PyException(Python.PyRef(Ptr{Python.CPython.PyObject} @0x00007f38c279e3a0), Python.PyRef(Ptr{Python.CPython.PyObject} @0x00007f38cc073e00), Python.PyRef(Ptr{Python.CPython.PyObject} @0x00007f38cc077b40))>
Python stacktrace:<error while printing stacktrace: Python.PyException(Python.PyRef(Ptr{Python.CPython.PyObject} @0x00007f38c279e3a0), Python.PyRef(Ptr{Python.CPython.PyObject} @0x00007f38c3024770), Python.PyRef(Ptr{Python.CPython.PyObject} @0x00007f38c3025ec0))>
Stacktrace:
  [1] pythrow()
    @ Python ~/.julia/packages/Python/HIdui/src/PyException.jl:26
  [2] macro expansion
    @ ~/.julia/packages/Python/HIdui/src/eval.jl:141 [inlined]
  [3] (::Python.var"#172#182")()
    @ Python ~/.julia/packages/Python/HIdui/src/init.jl:184
  [4] with_gil(f::Python.var"#172#182", c::Bool)
    @ Python ~/.julia/packages/Python/HIdui/src/gil.jl:10
  [5] with_gil
    @ ~/.julia/packages/Python/HIdui/src/gil.jl:9 [inlined]
  [6] macro expansion
    @ ~/.julia/packages/Python/HIdui/src/init.jl:180 [inlined]
  [7] (::Python.var"#169#178")()
    @ Python ~/.julia/packages/Requires/035xH/src/init.jl:11
  [8] __init__()
    @ Python ~/.julia/packages/Requires/035xH/src/init.jl:18
  [9] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:670
 [10] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base ./loading.jl:756
 [11] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:994
 [12] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:910
 [13] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:897
during initialization of module Python

$ python3
Python 3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
``

[cjdoris]

I couldn't reproduce this on MacOS or Windows. Could you update PyCall and Python and tell me the output of the following (in two separate sessions)?

using Python; Python.CONFIG
using PyCall; print(read(PyCall.depfile, String))

[PallHaraldsson]

Feel free to just close this as "user error", or to document this and/or make two different Python versions work. At least it's "documented" here, in case people look here. It's valuable to use PyCall.jl and Python.jl together I guess (assuming works, didn't go further than using, both orders). It's less useful/needed to support each using different Python.

I didn't realize I had 3.6 and 3.8 (python3 for the latter, but pip3 for the former, why I found out...).

What fixed this for me:

julia> ENV["PYTHON"] = "python3.8";
julia> Pkg.build("PyCall")

This was likely never a Linux issue, would have been on all platforms.

Before:

julia> print(read(PyCall.depfile, String))
const python = "/usr/bin/python3"
const libpython = "/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0"
const pyprogramname = "/usr/bin/python3"
const pyversion_build = v"3.6.9"
const PYTHONHOME = "/usr:/usr"

"True if we are using the Python distribution in the Conda package."
const conda = false

julia> Python.CONFIG
dlopenflags: 0x00000046
exepath: "/home/pharaldsson_sym/miniconda3/bin/python3.8"
libpath: "/home/pharaldsson_sym/miniconda3/lib/libpython3.8.so.1.0"
libptr: Ptr{Nothing} @0x0000000004997160
pyhome: "/home/pharaldsson_sym/miniconda3:/home/pharaldsson_sym/miniconda3"
pyhome_w: Int32[47, 104, 111, 109, 101, 47, 112, 104, 97, 114, 97, 108, 100, 115, 115, 111, 110, 95, 115, 121, 109, 47, 109, 105, 110, 105, 99, 111, 110, 100, 97, 51, 58, 47, 104, 111, 109, 101, 47, 112, 104, 97, 114, 97, 108, 100, 115, 115, 111, 110, 95, 115, 121, 109, 47, 109, 105, 110, 105, 99, 111, 110, 100, 97, 51, 0]
pyprogname: "/home/pharaldsson_sym/miniconda3/bin/python3.8"
pyprogname_w: Int32[47, 104, 111, 109, 101, 47, 112, 104, 97, 114, 97, 108, 100, 115, 115, 111, 110, 95, 115, 121, 109, 47, 109, 105, 110, 105, 99, 111, 110, 100, 97, 51, 47, 98, 105, 110, 47, 112, 121, 116, 104, 111, 110, 51, 46, 56, 0]
isstackless: false
isembedded: false
isinitialized: true
preinitialized: false
version: v"3.8.5-final+0"
isconda: true
condaenv: "/home/pharaldsson_sym/miniconda3"
pyplotautoshow: true
qtfix: true
sysautolasttraceback: true
inputhookrunning: false
ipythonintegration: true
isipython: false

[cjdoris]

Good to know it was a version issue, thanks.

I have just added a check that Python and PyCall are using the same version, and emit a warning if not. Grateful if you could check you see the warning?

[PallHaraldsson]

Yes, if I go back to libpython to being different I do get the warning, but only if I do using PyCall first. It's of course out of your hands in it follow. Then @stevengj might want to know about your package and put in a similar warning (copy your code).

This is however not really a huge concern for me, as I said, the important thing is that both can work together, as they do in the usual case.

Unlike PyCall, I see you have lots of code, which I guess slows down init, to support e.g. Pandas. There are other ways to speed it up besides dropping the code, but should i be optional? In an addon package?

and FYI: since I see you added trademark mark, I see at Python.org, that it's the correct one is actually "TM".

[cjdoris]

You should get a warning in either order. I use Requires.jl to detect if PyCall.jl was loaded after.

The large amount of code mainly comes from the Julia wrapper types and the general conversion system. The "helpers" (such as pandas support) are tiny. The slow init time comes from needing to compile C functions.

Python's trademark usage policy says to put "(R)" after the word Python. It's the logo that requires "TM".

[cjdoris]

FYI you can now set JULIA_PYTHONCALL_EXE=PYCALL to ensure compatability.