Opened 5 years ago

Last modified 5 weeks ago

#10884 confirmed defect

LNK2005 linker errors when wxUSE_STL and an external module uses std::vector<int>

Reported by: hajokirchhoff Owned by: vadz
Priority: normal Milestone: 3.1.0
Component: base Version: dev-latest
Keywords: wxUSE_STD_CONTAINERS Cc: troelsk@…
Blocked By: Blocking:
Patch: no

Description

This is with wxMSW, Visual Studio 2008, svn approx 10 months old.

Summary:

#define wxUSE_STL 1 prevents using std::vector<int> in non-wxwidgets modules when trying to link to wxWidgets dynamically. It causes several LNK2005 std::vector<int>::~vector<int> already defined in wxbase*.dll

Bug:

If #define wxUSE_STL 1, then wxbase*.dll exports all std::vector<T> members where T is short, int, double and a lot of others. This clashes with any module using std::vector<T> that attempts to link wxWidgets dynamically. I haven't tried static linking.

IOW, it is not possible to use std::vector<T> in my own code and dynamically link to wxWidgets when T is one of (int, double, short etc...)

Here are the details and a suggested fix:

The problem is dynarray.h, WX_DECLARE_BASE_ARRAY_2

#define _WX_DECLARE_BASEARRAY_2(T, name, predicate, classexp) \
classexp name : public std::vector<T> \

This will expand to

class WXDLLIMPEXP wxBaseArrayInt:public std::vector<int> ...

with WXDLLIMPEXP = __declspec(dllexport)

Experimenting with Visual Studio I found that if I dllexport a class that derives from a template, the linker will instantiate all template members and dllexport them as well.

So in this case all std::vector<int> members will be instantiated (code generated for them) and exported.

When a completely different module (DLL) uses the same template, the instantiated methods will clash with the exported ones from the wxbase*.dll and a linker error LNK2005 (function already defined in foo.obj) prevents linking.

Solution:

Instead of exporting the entire class, export only its members functions, so that the base class will not be exported.

Change DECLARE_BASEARRAY_2 to

#define _WX_DECLARE_BASEARRAY_2(T, name, predicate, classexp) \
class name : public std::vector<T> \
{ \
classexp name(); \
classexp size_t Count();

and so on.

This will expand to

class wxBaseArrayInt:public std::vector<int>
{
WXDLLIMPEXP wxBaseArrayInt();
WXDLLIMPEXP size_t Count();

I will prepare a fix and attach it to this ticket.

Attachments (3)

dynarray.patch download (17.1 KB) - added by hajokirchhoff 4 years ago.
dynarray.2.patch download (1.4 KB) - added by troelsk 15 months ago.
dll-diff.txt download (15.1 KB) - added by awi 6 weeks ago.
Debug DLL vs Release DLL (exported functions)

Download all attachments as: .zip

Change History (41)

comment:1 Changed 5 years ago by hajokirchhoff

Unfortunately the patch is a little bit more complicated than it might seem at first and also has the potential to break existing code.

The problem is that the original macro

#define  _WX_DECLARE_BASEARRAY(T, name, classexp)

expects the C++ keyword 'class' or 'struct' to be part of classexp. All through the wxWidgets sources you will see statements such as

and an array of class info objects
WX_DEFINE_USER_EXPORTED_ARRAY_PTR(wxClassInfo *, wxArrayClassInfo,
                                    class WXDLLIMPEXP_BASE);

I personally think, passing 'class WXDLLIMPEXP_BASE' as the classexp parameter isn't a good idea anyway, but this also prevents fixing the duplicate symbol linker error LNK2005.

The fix requires changing
   size_t Count(); to something    classexp size_t Count();
If the keyword 'class' is part of classexp, the fix won't work.

Unless someone has a good idea how to remove   'class' from a parameter macro 'class __declspec(dllexport)'   and do this in a macro definition, the only way to fix this problem is to get rid of the 'class' keyword in the macro parameter.

That means changing all instances of WX_DEFINE_USER_EXPORTED_ARRAY*


But without fixing this problem, wxUSE_STL prevents _all_ use of std::vector<int, short, float, double> in modules other than wxWidgets.

I've attached a patch for dynarray.h showing the fix in principle.

comment:2 Changed 5 years ago by hajokirchhoff

  • Patch set

comment:3 follow-up: Changed 5 years ago by vadz

  • Component changed from GUI-generic to base
  • Patch unset
  • Status changed from new to confirmed

AFAIR the macro was written like this because passing just WXDLLIMPEXP_BASE resulted in warnings in non-DLL build when this macro is empty and so the macro was called without enough parameters. OTOH I see that now we have wxARRAY_EMPTY which seems to take care of this (and also is redundant with wxEMPTY_PARAMETER_VALUE from wx/cpp.h...) so maybe it's not a problem any more.

In any case breaking the existing code using WX_DEFINE_USER_EXPORTED_ARRAY doesn't seem like a good idea as the error messages will undoubtedly be very unclear. This also prevents people from writing code compatible with both 2.8 and 3.0 which is bad as well. So the only solution I can see is to add another macro which would use expmode only before the methods and use it for wxArray{Short,Int,Double,Long,PtrVoid} definitions. I'd be glad to apply such a patch, of course, but the current one can't be used unfortunately, sorry.

comment:4 in reply to: ↑ 3 Changed 5 years ago by hajokirchhoff

Replying to vadz:

WX_DEFINE_USER_EXPORTED_ARRAY doesn't seem like a good idea as the error messages will undoubtedly be very unclear. This also prevents people from writing code compatible with both 2.8 and 3.0 which is bad as well. So the only solution I can see is to add another macro which would use expmode only before the methods and use it for wxArray{Short,Int,Double,Long,PtrVoid} definitions. I'd be glad to apply such a patch, of course, but the current one can't be used unfortunately, sorry.

Hm, not nice, but I don't see a better solution. I'll prepare a patch. Thanks.

comment:5 Changed 4 years ago by hajokirchhoff

  • Patch set

Okay, I did some more experimenting on this and came up with a way to remove the 'class' keyword from the classexp parameter. I prepend the token remove_class_keyword_ to the classexp parameter of the WX_DECLARE_BASE_ARRAY_2 and also define an empty preprocessor macro remove_class_keyword_class.

WX_DECLARE_BASE_ARRAY_2(T, name, predicate, classexp) \

WX_DECLARE_BASE_ARRAY_2_noclasskeyword(t, name, predicate, remove_class_keyword_##classexp)

I changed the original WX_DECLARE_BASE_ARRAY_2 to WX_DECLARE_BASE_ARRAY_2_noclasskeyword and can now change the class definition, so that it no longer exports the entire class, but exports its members only.

The only incompatibility left now is wxARRAY_EMPTY. This macro is used to avoid a preprocessor error when the classexp is empty. Example:

#define WX_DEFINE_USER_EXPORTED_ARRAY(T, name, expmode) \

WX_DEFINE_TYPEARRAY_WITH_DECL(T, name, wxBaseArrayPtrVoid, wxARRAY_EMPTY expmode)

If expmode is empty, wxARRAY_EMPTY will still get passed by the preprocessor. If it weren't, some compilers would complain.

However this construct adds a space between wxARRAY_EMPTY and expmode. This space prevents my workaround above from working.

The incompatibility is this: The order of wxARRAY_EMPTY and expmode needs to be reversed.

Instead of

(T, name, ..., wxARRAY_EMPTY expmode)

you have to use

(T, name, ..., expmode wxARRAY_EMPTY)

I hope that the fact that wxARRAY_EMPTY is used only inside dynarray.h - and nowhere else in wxMSW - means that it is an internal construct only and this incompatibility has no effect in practice.

If accepted this patch would solve the original problem in a completely backward compatible way with minimum fuss and minimum code repetition.

Changed 4 years ago by hajokirchhoff

comment:6 Changed 4 years ago by vadz

Thanks for your work on this! I don't think there is any problem with compatibility here.

Looking at the patch I don't understand the #ifdef NOT part however -- does it really need to be kept? It looks like the patch could be simplified further by just removing it and always using the new version of the macro, am I missing something?

comment:7 Changed 4 years ago by hajokirchhoff

Oops, sorry. #ifdef NOT is just my way of excluding a code block. I left the original code block intact for a while to compare it against the new version.

Yes, it can and should be removed. It does nothing.

Thanks

comment:8 Changed 4 years ago by vadz

  • Milestone set to 2.9.2

Looking at this again, while "remove class keyword" hack is ingenious, I'd like to be able to somehow avoid it... Would it be possible to modify the macros to take just the "expmode" without "class" in it in any case? I think I originally inserted "class" into this argument just to avoid the problems in static build (when "expmode" itself was empty) but as we had to use wxARRAY_EMPTY hack anyhow later, I don't see why do we need to keep it.

IOW would it be a problem to replace all occurrences of class expmode with wxARRAY_EMPTY expmode (or maybe something like wxARG_OR_EMPTY(expmode) where we'd have #define wxARE_OR_EMPTY(x) x wxEMPTY in wx/cpp.h) and insert class in the macros that need it? It looks to me like the code would be significantly simpler like this, even if it does demand more changes.

comment:9 Changed 4 years ago by hajokirchhoff

That's what I did in my first patch 14 months ago: replace all occurrences of class expmode. But it would break WX_DEFINE_USER_EXPORTED_ARRAY, which might be used elsewhere. You thought it wouldn't be a good idea to break that macro (see your first comment to my ticket).

I agree that this would be a cleaner and IMHO better way, but, as I said, it would either break WX_DEFINE_USER_EXPORTED_ARRAY or we would need to introduce another macro (lots of actually) WX_DEFINE_USER_EXPORTED_ARRAY_WITHOUT_CLASS, which would make the code even less readable than the "remove class keyword" hack.

comment:10 Changed 3 years ago by vadz

  • Milestone changed from 2.9.2 to 3.0

Sorry for yet another question about this but while preparing to apply the patch I couldn't help wondering if we need to export anything in wxUSE_STL case at all. I.e. I think that all methods of these classes are defined inline so would it be possible to just not use WXDLLIMPEXP_BASE in WX_DECLARE_USER_EXPORTED_BASEARRAY calls in wx/dynarray.h? AFAICS everything should still work then and nothing should be implicitly exported.

Am I missing anything here? If I had a test case I'd be able to try it myself but unfortunately I don't...

Thanks again!

comment:11 Changed 15 months ago by vadz

  • Milestone changed from 3.0 to 2.9.5

We probably should just apply the patch but if anybody can test if removing the dllexport keyword at all could be enough to fix it, it would be great.

comment:12 follow-up: Changed 15 months ago by troelsk

Will gladly test this, but, remove dllexport from where exactly?

PS: Related ticket #14990

comment:13 Changed 15 months ago by hajokirchhoff

Perhaps also related: #14741

comment:14 in reply to: ↑ 12 Changed 15 months ago by vadz

Replying to troelsk:

Will gladly test this, but, remove dllexport from where exactly?

I don't remember what was I thinking when I wrote it any more so sorry in advance if it's something stupid, but the comment:11 proposed removing it from WX_DECLARE_USER_EXPORTED_BASEARRAY. And as the fix for #14741 demonstrates, not exporting the classes at all does seem to solve a similar problem elsewhere, so I think we should remove them, perhaps only keeping them for the old (<= 7?) versions of MSVC.

Changed 15 months ago by troelsk

comment:15 Changed 15 months ago by troelsk

I tried removing dllexport thus, dynarray.2.patch, and it did not work for me, again got the linker error reported in ticket #14990, below. The original dynarray.patch fix it.

1>Linking...
1>wxmsw29ud_core.lib(wxmsw295ud_core_vc_custom.dll) : error LNK2005: "protected: __thiscall std::_Container_base_aux_alloc_empty<class std::allocator<class wxWindow *> >::~_Container_base_aux_alloc_empty<class std::allocator<class wxWindow *> >(void)" (??1?$_Container_base_aux_alloc_empty@V?$allocator@PAVwxWindow@@@std@@@std@@IAE@XZ) already defined in docview.obj
1>wxmsw29ud_core.lib(wxmsw295ud_core_vc_custom.dll) : error LNK2005: "protected: __thiscall std::_Container_base_aux_alloc_empty<class std::allocator<class wxWindow *> >::_Container_base_aux_alloc_empty<class std::allocator<class wxWindow *> >(class std::allocator<class wxWindow *>)" (??0?$_Container_base_aux_alloc_empty@V?$allocator@PAVwxWindow@@@std@@@std@@IAE@V?$allocator@PAVwxWindow@@@1@@Z) already defined in mysource.obj

comment:16 Changed 15 months ago by troelsk

I got that completely wrong sorry, please ignore comment above. Still do not know where to find that dllexport

comment:17 Changed 15 months ago by vadz

How exactly can I reproduce the problem? I've tried the example from #14990 and also just using vector<int> but I don't see any problems, everything links just fine in release DLL build:

  • samples/minimal/minimal.cpp

    diff --git a/samples/minimal/minimal.cpp b/samples/minimal/minimal.cpp
    index a78e462..9572cd8 100644
    a b void MyFrame::OnQuit(wxCommandEvent& WXUNUSED(event)) 
    183183    Close(true); 
    184184} 
    185185 
     186#include <vector> 
     187 
    186188void MyFrame::OnAbout(wxCommandEvent& WXUNUSED(event)) 
    187189{ 
     190#if wxUSE_STL 
     191    const wxWindowList wndList = GetChildren(); 
     192    std::vector<int> nums(10); 
     193#else 
     194#error !wxUSE_STL 
     195#endif 
     196 
    188197    wxMessageBox(wxString::Format 
    189198                 ( 
    190199                    "Welcome to %s!\n" 

So what problem are we actually trying to solve here?

comment:18 Changed 15 months ago by hajokirchhoff

Hi Vadim,
we recently had a fix that essentially removed the _declspec(dllexport) from the STD compatible wxArray classes. The (old) wxArray classes inherit from std::vector/std::list, if wxUSE_STL is defined. Until recently the resulting classes would be exported, but that fix removed dllexport and made them template classes only. The ticket is #14741 (I believe).

It is quite possible that that fix also resolves this problem. This problem here is caused by the fact that wxDECLARE_ARRAY will create a class derived from vector<int> and export that class. This will at the same time instantiate vector<int> itself and also export it. That causes the linker error I described in this ticket.

My guess is that since #14741 removes the dllexport from the wxDECLARE_ARRAY, the resulting class is still a template class, will not be exported and will no longer clash with other instances of vector<int>.

To verify that you could undo #14741 and see it the problem occurs then.

Or try to reproduce the bug as follows:
Compile wxWidgets DLLs with wxUSE_STL
Modify a sample to use vector<int> somewhere.
Try to link the sample with the wxWidgets DLLs.

comment:19 Changed 15 months ago by vadz

  • Resolution set to fixed
  • Status changed from confirmed to closed

Well, this is exactly what the patch in comment:17 does, or am I missing something?

So I'm tentatively closing it, please reopen if the problem can still be reproduced.

Thanks!

comment:20 Changed 12 months ago by troelsk

  • Resolution fixed deleted
  • Status changed from closed to reopened

please reopen if the problem can still be reproduced.

It still can, in a project of mine. I have been using the dynarray.patch since.
Just tried un-applying it: then again I get the error reported in #14990.
(using Visual Studio 2008, DLL target, wxUSE_STL + wxUSE_STD_CONTAINERS + WXUSINGDLL)

comment:21 Changed 12 months ago by vadz

Are you still getting the same errors as in comment:15?

Any idea about how can I reproduce this?

comment:22 Changed 12 months ago by troelsk

Any idea about how can I reproduce this?

No. Because I see it only in a dll compile here I tried sprinkling the dll sample with wxWindowList's, instantiating, exporting, without luck so far. Will try again if I get an idea.

Are you still getting the same errors as in comment:15?

Almost,

1>wxmsw29ud_core.lib(wxmsw295ud_core_vc_custom.dll) : error LNK2005: "protected: __thiscall std::_Container_base_aux_alloc_empty<class std::allocator<class wxWindow *> >::~_Container_base_aux_alloc_empty<class std::allocator<class wxWindow *> >(void)" (??1?$_Container_base_aux_alloc_empty@V?$allocator@PAVwxWindow@@@std@@@std@@IAE@XZ) already defined in mysourcefile.obj
1>wxmsw29ud_core.lib(wxmsw295ud_core_vc_custom.dll) : error LNK2005: "protected: __thiscall std::_Container_base_aux_alloc_empty<class std::allocator<class wxWindow *> >::_Container_base_aux_alloc_empty<class std::allocator<class wxWindow *> >(class std::allocator<class wxWindow *>)" (??0?$_Container_base_aux_alloc_empty@V?$allocator@PAVwxWindow@@@std@@@std@@IAE@V?$allocator@PAVwxWindow@@@1@@Z) already defined in mysourcefile.obj

comment:23 Changed 12 months ago by troelsk

To summarize:

  • I experience the linking problem only with wxWindowList, not with std::vector<int> as the OP does
  • not reproducable in the samples

I suspect that this declaration change would fix the problem,

#if wxUSE_STD_CONTAINERS
class wxWindowList : public wxVector<wxWindow*> {};
#else
WX_DECLARE_LIST_3(wxWindow, wxWindowBase, wxWindowList, wxWindowListNode, class WXDLLIMPEXP_CORE);
#endif

BUT this breaks wx build in many places as std::vector has no Find/Append/GetCount/GetFirst/DeleteObject methods, quite a bit of work to STL'ize wx code using wxWindowList.
Is it worth considiring going down this path??

comment:24 Changed 12 months ago by hajokirchhoff

If you have this problem only with wxWindowList, the problem may be in your code. Are you using a vector<wxWindow*> or something somewhere?

comment:25 Changed 12 months ago by troelsk

  • Cc troelsk@… added

No, no vector<wxWindow*> anywhere.
The problem seems to be inside wx, but is not reproducible in the samples :-(

comment:26 Changed 11 months ago by troelsk

  • Resolution set to invalid
  • Status changed from reopened to closed

I narrowed it down to this one line in my dll,

    const wxWindowList wndList(frame.GetChildren()); // make a copy of the window list -> linker error in wx trunk+wxUSE_STL+DLL

Making it a reference (not what I want here) makes it link

    const wxWindowList& wndList(frame.GetChildren()); // no dtor called, linker is happy

Still not reproducible in the sample. Maybe the compiler is broken, closing ticket again.
(Will make myself a workaround using a simple array of wxWindow pointers)

comment:27 Changed 6 weeks ago by troelsk

  • Keywords wxUSE_STD_CONTAINERS added; LNK2005 _WX_DECLARE_BASE_ARRAY stl removed
  • Milestone 2.9.5 deleted
  • Patch unset
  • Priority changed from high to normal
  • Resolution invalid deleted
  • Status changed from closed to reopened
  • Version set to dev-latest

Since r76121 today wx itself fails to build in configuration "DLL Debug" with wxUSE_STD_CONTAINERS=1.

Reproducible like this:

  • clean svn trunk checkout
  • copy setup0.h -> setup.h, changing wxUSE_STD_CONTAINERS from 0 to 1
  • build "DLL Debug" with MSVC 2008 or 2012
  • link error:

MSVC 2008 Pro SP1, wx_vc9.sln

1>------ Build started: Project: stc, Configuration: DLL Debug Win32 ------
1>Linking...
1>wxbase31ud.lib(wxbase310ud_vc_custom.dll) : error LNK2005: "public: int & thiscall std::vector<int,class std::allocator<int> >::operator[](unsigned int)" (??A?$vector@HV?$allocator@H@std@@@std@@QAEAAHI@Z) already defined in wxscintillad.lib(AutoComplete.obj)
1> Creating library ..\..\lib\vc_dll\wxmsw31ud_stc.lib and object ..\..\lib\vc_dll\wxmsw31ud_stc.exp
1>..\..\lib\vc_dll\wxmsw310ud_stc_vc_custom.dll : fatal error LNK1169: one or more multiply defined symbols found
1>Build log was saved at "file://c:\svn\wx30clean\build\msw\vc_mswuddll\stc\BuildLog.htm"
1>stc - 2 error(s), 0 warning(s)
========== Build: 0 succeeded, 1 failed, 22 up-to-date, 0 skipped ==========

MSVC 2012 Express, wx_vc11.sln

1>------ Build started: Project: stc, Configuration: DLL Debug Win32 ------
1>wxbase30ud.lib(wxbase30ud_vc_custom.dll) : error LNK2005: "public: int & thiscall std::vector<int,class std::allocator<int> >::operator[](unsigned int)" (??A?$vector@HV?$allocator@H@std@@@std@@QAEAAHI@Z) already defined in wxscintillad.lib(PositionCache.obj)
1> Creating library ..\..\lib\vc_dll\wxmsw30ud_stc.lib and object ..\..\lib\vc_dll\wxmsw30ud_stc.exp
1>..\..\lib\vc_dll\wxmsw30ud_stc_vc_custom.dll : fatal error LNK1169: one or more multiply defined symbols found
========== Build: 0 succeeded, 1 failed, 22 up-to-date, 0 skipped ==========

comment:28 Changed 6 weeks ago by vadz

  • Milestone set to 3.1.0
  • Status changed from reopened to confirmed

OK, I see this too but I have absolutely no idea about what is going on here :-( Worse, I don't see what part of r76121 exactly triggered it. I thought it was the addition of std::vector<int> to AutoComplete.h but reverting this didn't help. I'll try to return to this later but in the meanwhile any help/ideas would be welcome...

comment:29 Changed 6 weeks ago by awi

It seems that only "Debug DLL" configuration is affected by this issue. "Release DLL" can be built with no problems.
Apparently, in the Debug mode there are exported more functions then in the Release mode (list here attachment:ticket:10884:dll-diff.txt ).

Linking is successful if in the "base" project for "Debug DLL" target there is defined (and set) macro _ITERATOR_DEBUG_LEVEL=1 .

  • build/msw/wx_vc11_base.vcxproj

    old new  
    313313      <AdditionalOptions>/MP %(AdditionalOptions)</AdditionalOptions> 
    314314      <Optimization>Disabled</Optimization> 
    315315      <AdditionalIncludeDirectories>$(OutDir)$(wxIncSubDir);..\..\include;..\..\src\tiff\libtiff;..\..\src\jpeg;..\..\src\png;..\..\src\zlib;..\..\src\regex;..\..\src\expat\lib;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories> 
    316       <PreprocessorDefinitions>WIN32;_USRDLL;DLL_EXPORTS;_DEBUG;_CRT_SECURE_NO_DEPRECATE=1;_CRT_NON_CONFORMING_SWPRINTFS=1;_SCL_SECURE_NO_WARNINGS=1;__WXMSW__;_UNICODE;WXBUILDING;wxUSE_GUI=0;WXMAKINGDLL_BASE;wxUSE_BASE=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> 
     316      <PreprocessorDefinitions>WIN32;_USRDLL;DLL_EXPORTS;_DEBUG;_CRT_SECURE_NO_DEPRECATE=1;_CRT_NON_CONFORMING_SWPRINTFS=1;_SCL_SECURE_NO_WARNINGS=1;__WXMSW__;_UNICODE;WXBUILDING;wxUSE_GUI=0;WXMAKINGDLL_BASE;wxUSE_BASE=1;_ITERATOR_DEBUG_LEVEL=1;%(PreprocessorDefinitions)</PreprocessorDefinitions> 
    317317      <ExceptionHandling>Sync</ExceptionHandling> 
    318318      <BasicRuntimeChecks>EnableFastChecks</BasicRuntimeChecks> 
    319319      <RuntimeLibrary>MultiThreadedDebugDLL</RuntimeLibrary> 

Or maybe this macro should be set conditionally if wxUSE_STD_CONTAINERS=1 ?

Changed 6 weeks ago by awi

Debug DLL vs Release DLL (exported functions)

comment:30 Changed 6 weeks ago by vadz

Thanks awi, this at least gives us a workaround. But I'm not sure if we don't have to compile the user code using the DLL with _ITERATOR_DEBUG_LEVEL=1 too then though -- and that would, of course, be bad. BTW, we'd also have to use _HAS_ITERATOR_DEBUGGING for VC9, as _ITERATOR_DEBUG_LEVEL seems to only have appeared in VC10.

And I still have absolutely no idea why does it help, looking at <vector> contents, _ITERATOR_DEBUG_LEVEL checks only control what's going on inside operator[](size_t) implementation, but not why is it exported or not. Could it be related to inlining somehow?

Finally, to deepen the mystery, everything links just fine with VC12 (2013), even though nothing much seems to have changed since VC11...

comment:31 Changed 6 weeks ago by vadz

Unfortunately I don't seem to be able to make it work with _HAS_ITERATOR_DEBUGGING=0 for VC9 projects.

comment:32 Changed 6 weeks ago by awi

Really strange issue, indeed.
Unfortunately, I don't have VS 2008 at hand and I can't test VC9 projects.
It's claimed in MSDN MSDN that _ITERATOR_DEBUG_LEVEL=1 is an equivalent to _HAS_ITERATOR_DEBUGGING=0 and _SECURE_SCL=1 .
Maybe both macros should be set under VS2008?

comment:33 Changed 5 weeks ago by awi

  1. I did some experiments and it appeared that these linker errors are caused in this particular case by instantiation of the following class members in the Scintilla library:
  • src/stc/scintilla/lexers/LexLaTeX.cxx

    old new  
    4747 
    4848class LexerLaTeX : public LexerBase { 
    4949private: 
    50         vector<int> modes; 
     50    vector<int> modes;  // ! 
    5151        void setMode(int line, int mode) { 
    5252                if (line >= static_cast<int>(modes.size())) modes.resize(line + 1, 0); 
    5353                modes[line] = mode; 
  • src/stc/scintilla/src/AutoComplete.h

    old new  
    2121        char separator; 
    2222        char typesep; // Type seperator 
    2323        enum { maxItemLen=1000 }; 
    24         std::vector<int> sortMatrix; 
     24        std::vector<int> sortMatrix; // ! 
    2525 
    2626public: 
    2727 
  • src/stc/scintilla/src/PositionCache.h

    old new  
    153153        int lineEnd; 
    154154        int posLineStart; 
    155155        int nextBreak; 
    156         std::vector<int> selAndEdge; 
     156        std::vector<int> selAndEdge; // ! 
    157157        unsigned int saeCurrentPos; 
    158158        int saeNext; 
    159159        int subBreak; 

But these errors would happen for any third-party library containing instantiation of std::vector<int>, etc.

  1. It seems that _ITERATOR_DEBUG_LEVEL macro only masks the problem. In the Release mode IDL is set to 0 by default what means that there is no checking (for maximum performance) and this is why there are no errors reported in this case. (See this discussion)
  1. Generally, using STL instantiation in DLLs is discouraged so maybe the best option would be just to enforce wxUSE_STD_CONTAINERS=0 when DLL is built?

comment:34 Changed 5 weeks ago by vadz

Thanks a lot for your continued help!

Unfortunately I still don't understand what exactly is going on here. I understand the _ITERATOR_DEBUG_LEVEL problem and the resulting ABI issues, but it should (and AFAICS is) the same in wxscintilla and wxbase. Also, it doesn't look like the operator[] creating the problem is actually being exported from wxbase DLL: if it were, we presumably wouldn't have any problem, actually, as it wouldn't conflict with it being defined in wxscintilla static lib.

And we really can't avoid instantiating standard containers in the DLL, wxUSE_STD_CONTAINERS changes the ABI, so it must be the same for the user code and DLL builds...

comment:35 follow-up: Changed 5 weeks ago by awi

It seems to me that at least for MSVC it wouldn't be possible to avoid these collisions of STL container instantiations in wx and external libraries.
Maybe an acceptable workaround would be to build wrappers for basic types and instantiate containers for these wrappers? I mean something like this:

  • include/wx/dynarray.h

    old new  
    834834// Some commonly used predefined base arrays 
    835835// ---------------------------------------------------------------------------- 
    836836 
     837#if defined(_USRDLL) && defined(_MSC_VER) 
     838class wxInt 
     839{ 
     840public: 
     841    wxInt() {} 
     842    wxInt(int v) :m_val(v) {} 
     843    wxInt& operator=(int v) { m_val = v; return *this; } 
     844    wxInt& operator-=(int v) { m_val -= v; return *this; } 
     845    wxInt& operator+=(int v) { m_val += v; return *this; } 
     846    wxInt operator++(int) { wxInt res = *this; m_val++; return res; } 
     847    operator int() const { return m_val; } 
     848//    int* operator&() { return &m_n; } 
     849private: 
     850    int m_val; 
     851}; 
     852#else 
     853typedef int wxInt; 
     854#endif 
     855 
    837856WX_DECLARE_USER_EXPORTED_BASEARRAY(const void *, wxBaseArrayPtrVoid, 
    838857                                   WXDLLIMPEXP_BASE); 
    839858WX_DECLARE_USER_EXPORTED_BASEARRAY(char, wxBaseArrayChar, WXDLLIMPEXP_BASE); 
    840859WX_DECLARE_USER_EXPORTED_BASEARRAY(short, wxBaseArrayShort, WXDLLIMPEXP_BASE); 
    841 WX_DECLARE_USER_EXPORTED_BASEARRAY(int, wxBaseArrayInt, WXDLLIMPEXP_BASE); 
     860WX_DECLARE_USER_EXPORTED_BASEARRAY(wxInt, wxBaseArrayInt, WXDLLIMPEXP_BASE); 
    842861WX_DECLARE_USER_EXPORTED_BASEARRAY(long, wxBaseArrayLong, WXDLLIMPEXP_BASE); 
    843862WX_DECLARE_USER_EXPORTED_BASEARRAY(size_t, wxBaseArraySizeT, WXDLLIMPEXP_BASE); 
    844863WX_DECLARE_USER_EXPORTED_BASEARRAY(double, wxBaseArrayDouble, WXDLLIMPEXP_BASE); 
     
    10191038// ---------------------------------------------------------------------------- 
    10201039 
    10211040WX_DEFINE_USER_EXPORTED_ARRAY_SHORT(short, wxArrayShort, class WXDLLIMPEXP_BASE); 
    1022 WX_DEFINE_USER_EXPORTED_ARRAY_INT(int, wxArrayInt, class WXDLLIMPEXP_BASE); 
     1041WX_DEFINE_USER_EXPORTED_ARRAY_INT(wxInt, wxArrayInt, class WXDLLIMPEXP_BASE); 
    10231042WX_DEFINE_USER_EXPORTED_ARRAY_DOUBLE(double, wxArrayDouble, class WXDLLIMPEXP_BASE); 
    10241043WX_DEFINE_USER_EXPORTED_ARRAY_LONG(long, wxArrayLong, class WXDLLIMPEXP_BASE); 
    10251044WX_DEFINE_USER_EXPORTED_ARRAY_PTR(void *, wxArrayPtrVoid, class WXDLLIMPEXP_BASE); 

Proof of concept shown that there are no conflicts between wx and scintilla and everything seemed to work fine. One drawback of this approach I see is that there is not possible to use pointers to the items in collections ('address of' operator shouldn't be overloaded in case of using STL).
I can't tell if it is a serious limitation or not.

comment:36 in reply to: ↑ 35 ; follow-up: Changed 5 weeks ago by hajokirchhoff

Hi, sorry if my comment does not apply, I only skimmed the previous comments, but...

Replying to awi:

Maybe an acceptable workaround would be to build wrappers for basic types and instantiate containers for these wrappers? I mean something like this:

+#if defined(_USRDLL) && defined(_MSC_VER)
+class wxInt

This would be a very, very bad idea in my opinion. My experience is that sooner or later wrappers like this will cause more problems than they solve.

IMHO wxWidgets should never export a template derived container, at least not for basic datatypes. I see only two clean choices:

a) export wxArrayInt, but do not derive it from std::vector (as this would instantiate and export std::vector<int> as well). This means we would have to write a std::vector compatible wxArrayInt class, which has the same interface but does not actually use the std::vector template. Not good but possible.

  • or -

b) make wxArrayInt a template only class and do not export it at all. This should work fine. It should not be neccessary to _declspec(dllimport/dllexport) this class if it is a true template class. This is the solution I would prefer. It should not be neccessary to export this at all.

Solution a) would avoid the linker error as wxArrayInt would no longer cause std::vector<int> to be exported.
Solution b) would avoid the linker error as wxArrayInt would now be a true template class itself and would not be exported at all, but rather instantiated everywhere it is used. And the linker would clean up duplicate template instantiations automatically.

comment:37 in reply to: ↑ 36 Changed 5 weeks ago by awi

I agree with you (several posts ago I voted for setting wxUSE_STD_CONTAINERS=0 for DLL target) but it seems (see previous posts) that resignation neither from instantiation of std::vector family nor from exporting is not an option here (ABI requirements).
This is why so desperate solution was proposed :)

comment:38 Changed 5 weeks ago by vadz

We already have wxVector<> but this is not a solution. We do want to allow people use real, std::vector<>s, this is the whole point of wxUSE_STD_CONTAINERS: to allow interoperability with the application code (and/or other libraries) using the standard classes. And we can't avoid exporting wxArrayInt from the DLL, at least I don't see how would this be possible.

In principle, there is absolutely nothing wrong with what we're doing, I'm pretty sure we're just hitting some mysterious compiler bug (mysterious because I still have no idea how does the problem concretely arise, I think we'd need to look at the assembly generated by the compiler to see where exactly does it reference this operator[] and why does it use a non-DLL-exported version) and we "just" need some workaround for it...

Note: See TracTickets for help on using tickets.