Opened 6 years ago

Closed 2 years ago

#15588 closed enhancement (fixed)

Migration from Subversion to Git

Reported by: bpetty Owned by: robind
Priority: normal Milestone: 3.2.0
Component: project infrastructure Version:
Keywords: Cc: bpetty
Blocked By: Blocking:
Patch: no

Description

This is a tracking ticket for the migration off of Subversion, and over to using strictly git for version control.

We've only briefly discussed this on wx-dev occasionally, so I want to keep this ticket open for more discussion still, however, note that we really have already been making the migration for a while now in small increments. You may have noticed that our GitHub mirror has been running for about two and a half years now, and is already seeing quite a bit of use.

This migration is a delicate operation, and requires a fairly specific set of steps in the right order. We haven't been, and don't plan to rush any of this all at once. We've been on this road for a couple years obviously, and the rest of this could easily still take another couple years, please be patient.

Here's a basic outline of the remaining steps:

1. Upgrade Trac

We're still sitting back at version 0.11.5 currently, and git support within Trac wasn't official until 1.0. We've been hung up on this step for a long time now, but it really mostly just comes down to Robin having the time to do this (in fact, most of this migration hangs on Robin's free time to do this as he manages Trac and SVN currently).

2. Clone GitHub Repos to Trac

We will still want the ability to browse code on Trac, and hopefully maintain existing links to source code within the Trac browser. This will require a local clone running on Trac.

We could use this as the official repository with our own access controls, and mirror to GitHub, or simply use GitHub's access control layer to maintain team access and mirror back to Trac. This still needs some research as there could be many advantages to notification systems implemented directly on the Trac server and more fine-tuned control over repository protection that GitHub can't give us. Keep in mind that accepting GitHub pull requests if we do eventually (and we have no plans to initially still) has no effect on this decision.

Ideally, I believe we will likely end up leaving the read-only SVN repository available on Trac as it is now for archival purposes (leaving existing pegged revision URLs working), while adding the git repo as an additional source code base, but this is still undecided as well.

Note that even after we have the git repos on Trac, we can still continue to only mirror commits from SVN to git while we continue more training efforts for developers and committers who still haven't had the opportunity to really learn how to use git. This could happen depending on how comfortable we feel at this point with the team's experience.

3. Configure Git Notification Systems

The wx-commits and wx-commits-diffs mailing lists will need to be switched to provide git commit notifications. This will likely be as simple as moving over the post-commit hooks and switching out diff commands, but we'll see when we get here.

Our IRC notifications in #wxwidgets are actually already announcing git commits through the GitHub mirror, so we're set there.

4. Configure Trac Workflow Hooks

Automatic ticket mentions and closures are configured to watch SVN commits, this should also be fairly simple to switch over to git hooks once the repos are available.

5. Schedule the Switch

Anything beyond this point depends on the project flipping the switch to put the Subversion repos into read-only mode, and to start manually pushing patches up to the authoritative git repo. Again, this only happens when the entire team is comfortable with making the switch.

6. Update Contributor Documentation

The following documents will need updated with new instructions:

Topics Not Being Discussed Here

To keep this ticket as concise as possible, I would like to encourage everyone to avoid discussing related, but not directly related topics such as:

  • GitHub Pull Requests
  • Review Board vs Gerrit
  • Other Issue Trackers like GitHub Issues (we're locked on Trac, that's not changing)
  • Travis CI or other Continuous Integration Systems

Please move those discussions to the wx-dev mailing list along with any other topics not directly related to the migration from Subversion to Git.

Change History (30)

comment:1 Changed 6 years ago by bpetty

One more additional step after (5):

  • Configure EOL properties - see #15584

comment:2 follow-up: Changed 6 years ago by vaclavslavik

One more step:

  • Update the redirection rule for ^wxxrc$ in wxWebSite’s .htaccess file (currently points to http://svn.wxwidgets.org/svn/wx/wxWidgets/trunk/misc/schema/xrc_schema.rnc).

comment:3 Changed 6 years ago by vadz

  • Milestone changed from future to 3.2.0

I'd really like to do this in 3.2 time frame. Robin, please contact me if I can help with this in any way. TIA!

comment:4 Changed 6 years ago by robind

I'm working on getting caught up on wx things over the holidays. If there is any time left over after that then I'll try upgrading Trac too.

comment:5 follow-up: Changed 6 years ago by ghostvoodooman

Is there a possibility for me to be notified like I am receiving notification of commits by e-mail from wx-commits@ and wx-commits-diffs@ (from SVN server)? I need it because I have my own fork at github of wxWidgets intended for Intel Compiler with cutting-edge performance of result code. Though I am using github for years for fetching changes in wxWidgets github, and for pushing my fork to github, but this is done by command-line interface, but I am not so familiar with github in meaning of github's web site/GUI. Is there such feature to be notified of each commit? (preferably not only of each push which encapsulates more commits at once).

Any suggestion is welcome.

TIA!

comment:6 in reply to: ↑ 5 ; follow-up: Changed 6 years ago by bpetty

Replying to ghostvoodooman:

Is there a possibility for me to be notified like I am receiving notification of commits by e-mail from wx-commits@ and wx-commits-diffs@ (from SVN server)?

I'm not sure exactly what you're asking, but these mailing lists will continue to send notifications about changes just as they always have even after this migration (as mentioned in the ticket already). You can continue to use the same list as you might already be doing.

comment:7 in reply to: ↑ 6 Changed 6 years ago by ghostvoodooman

Replying to bpetty:

[...]
(as mentioned in the ticket already)

ouch, mea culpa, it is the paragraph #4 in the ticket. I must admit I have not read the whole ticket, since it is fairly long. For future, I will read at least headlines of all paragraphs. Sorry for the noise and your time.

comment:8 Changed 5 years ago by bpetty

Just FYI: Step 1 is done now. Trac has been upgraded to 1.0.1 as of now.

comment:9 Changed 5 years ago by bpetty

Step 2 is now mostly done.

The git repos just added to Trac are currently configured to just mirror the GitHub repos, and have no direct read or write access. It's looking like we might just make the GitHub repos the authoritative source, and push commits there.

We have stopped git-svn sync on the Phoenix repository, and the SVN repo for Phoenix will be made read-only shortly here making it the first repository to go git-only. This will be our trial run before doing this with the main repositories.

Related wx-dev discussion:
https://groups.google.com/forum/#!topic/wx-dev/YtYRyPZNWiQ

comment:10 Changed 5 years ago by bpetty

Another step:

comment:11 Changed 5 years ago by bpetty

Announcement has been made regarding the switch for the primary wxWidgets repository, which is scheduled for Feb 20th:

https://groups.google.com/d/msg/wx-dev/8-J8QCCHzlw/eWLA_gHmY6cJ

comment:12 in reply to: ↑ 2 Changed 5 years ago by bpetty

Replying to vaclavslavik:

  • Update the redirection rule for ^wxxrc$ in wxWebSite’s .htaccess file (currently points to http://svn.wxwidgets.org/svn/wx/wxWidgets/trunk/misc/schema/xrc_schema.rnc).

This is now done, pointing to:
https://raw.githubusercontent.com/wxWidgets/wxWidgets/master/misc/schema/xrc_schema.rnc

comment:13 Changed 5 years ago by bpetty

The following docs have now been updated:

comment:14 follow-up: Changed 5 years ago by bpetty

The wx-commits and wx-commits-diffs mailing lists are both receiving commit notifications for the wxWidgets and website repositories.

The wx-commits-diffs is no longer seeing full source code diffs though. In the interest of sticking with GitHub maintained email hooks though, we might just retire wx-commits-diffs in favor of only offering wx-commits notifications.

Additionally, the BuildSVN.txt instructions have been updated and moved to BuildGit.txt.

comment:15 follow-up: Changed 5 years ago by bpetty

Vadim: Have any ideas or preferences on how we should continue maintaining the 3rd party vendor libraries?

http://trac.wxwidgets.org/browser/svn-wx/wxWidgets/vendor/

comment:16 Changed 5 years ago by bpetty

Also note that none of the SVN hooks are active now:
http://trac.wxwidgets.org/browser/svn-wx/wxWidgets/trunk/misc/scripts/svn/hooks

comment:18 in reply to: ↑ 15 ; follow-up: Changed 5 years ago by vaclavslavik

Replying to bpetty:

Vadim: Have any ideas or preferences on how we should continue maintaining the 3rd party vendor libraries?

Although not Vadim, I think there aren’t that many options: either submodules or SVN-style vendor with subtree merges.

Of these dependencies, libtiff and expat use CVS (really) upstream, although expat has a GIT mirror right at the SF.net page (is that a SF.net thing or their own?). libpng and zlib use git upstream, so submodules would be easy for them; libtiff and expat would require mirrors.

FWIW, I get occasional compilation questions with (relatively low-volume) WinSparkle project due to people not following instructions and checking out submodules. It’s probably safe to say that not using them is more fool-proof and easier for wx users (which is a concern due to long release cycles).

wx also still applies a number of fixes/changes on the third party libraries, right? That would make submodules more hassle too.

I’d say go with vendor branches and merge them into the main tree, SVN-style. I do it with Poedit (see https://github.com/vslavik/poedit/branches/all; the primary motivation was the constant need to patch gettext’s build system) and it works reasonably well. The vendor/foo branches track upstream releases (i.e. unpack the tarball, git add --all) and are merged back to the master with git merge -s subtree -Xsubtree=deps/foo vendor/foo.

comment:19 in reply to: ↑ 18 ; follow-up: Changed 5 years ago by bpetty

Replying to vaclavslavik:

FWIW, I get occasional compilation questions with (relatively low-volume) WinSparkle project due to people not following instructions and checking out submodules. It’s probably safe to say that not using them is more fool-proof and easier for wx users (which is a concern due to long release cycles).

This has been my experience as well, and it seems like this is a common reason I've seen a lot of projects just avoid submodules altogether. Forgetting (or not even knowing it's necessary) to run submodule init and update certainly can lead to obscure bugs being reported due to out of sync dependencies.

I've also seem some really weird behavior you have to watch out for when checking out branches that never contained a submodule versus the one you were on that did. Tends to make it more difficult to maintain stable branches, or just do testing (especially if you want to make use of git-bisect).

wx also still applies a number of fixes/changes on the third party libraries, right? That would make submodules more hassle too.

I'm fairly certain there's custom changes, although, I don't see how that would make submodules any more difficult or any different at all really. We'd just point it at one of our own branches on one of our own repos (which we could maintain through subtree merges as well) rather than directly at the upstream's repo. We would do that anyway, even when we don't have changes.

I’d say go with vendor branches and merge them into the main tree, SVN-style. I do it with Poedit (see https://github.com/vslavik/poedit/branches/all; the primary motivation was the constant need to patch gettext’s build system) and it works reasonably well. The vendor/foo branches track upstream releases (i.e. unpack the tarball, git add --all) and are merged back to the master with git merge -s subtree -Xsubtree=deps/foo vendor/foo.

Just to be clear, you apply all custom patches directly to the deps/foo directory on the main branches, and the vendor branch remains an exact copy from upstream, right?

comment:20 in reply to: ↑ 19 ; follow-up: Changed 5 years ago by vaclavslavik

Replying to bpetty:

I'm fairly certain there's custom changes, although, I don't see how that would make submodules any more difficult or any different at all really.

It means you can’t use pristine upstream (or mirror of) and have to maintain custom changes in multiple places; any modifications to the deps has to be done with two commits (one with the change, one to update the submodule) too. I don’t see much advantage to that, it gets nightmarish pretty quickly. But it’s certainly possible, yes.

Just to be clear, you apply all custom patches directly to the deps/foo directory on the main branches, and the vendor branch remains an exact copy from upstream, right?

Yes.

comment:21 in reply to: ↑ 20 Changed 5 years ago by bpetty

Replying to vaclavslavik:

Replying to bpetty:

I'm fairly certain there's custom changes, although, I don't see how that would make submodules any more difficult or any different at all really.

It means you can’t use pristine upstream (or mirror of) and have to maintain custom changes in multiple places; any modifications to the deps has to be done with two commits (one with the change, one to update the submodule) too. I don’t see much advantage to that, it gets nightmarish pretty quickly. But it’s certainly possible, yes.

I just assume that if submodules were to be used, you still always want to have full control over where they point since it's literally part of the versioned code history of your repo, so even when you don't have custom changes, you still fork the upstream repo, and and maintain a copy within the project that the submodule points to instead. In that case, it's the same two edits (push update to fork, and submodule update) regardless of whether you have custom changes or not.

Anyway, I'm still in full agreement with you that using submodules at all just tends to cause more problems than it's worth. Even forking a non-modified upstream source is purely an attempt to avoid one small subset of problems caused by using submodules.

Just to be clear, you apply all custom patches directly to the deps/foo directory on the main branches, and the vendor branch remains an exact copy from upstream, right?

Yes.

I like this approach, but I tend to believe that if you're going to add an orphaned branch to your repo, there's no reason not to leave it in a separate repository (i.e. a fork of the upstream) where it's easier to propose patches and pull requests back upstream from the same repo that's maintaining any custom changes we make. Although, that means making changes in two repositories for a single custom change again, somewhat like using a submodule (except, without the headaches for users not touching the vendor lib). In reality though, we don't make those changes often, and should be discouraged from doing so anyway.

There's another approach that's easier than either of these though. If the vendor branch really is an exact copy of upstream, there's actually no reason to add it to the repo at all since you can use git subtree (not the same as a merge using the subtree strategy) with the upstream's repo directly (assuming there is an upstream repo using git):

$ git remote add --no-tags -f libpng git://git.code.sf.net/p/libpng/code
$ git subtree add --prefix deps/libpng libpng libpng-1.6.15-signed --squash

You can still make custom commits directly in the same directory (just like with your subtree merge approach), but the vendor branch isn't necessary, and updates are as simple as this (note that I pull 1.6.16 instead of 1.6.15):

$ git subtree pull --prefix deps/libpng libpng libpng-1.6.16-signed --squash

Atlassian has a good blog post explaining how this works.

Another added benefit to this approach is that git subtree also knows how to extract your custom changes if you want to pull them back out into a fork of upstream in order to prepare them as a patch to be submitted back upstream. So that provides some middle ground for someone like me that would like an easy way to push changes back upstream.

Last edited 5 years ago by bpetty (previous) (diff)

comment:22 Changed 5 years ago by bpetty

Looks like Vadim already maintains a git-cvsimport mirrors of both expat and libtiff:

Although SF.net doesn't just enable git by default, and it looks like the expat git repos are kept up to date, so either it's a built in mirror capability SF.net provides (that works the same way VZ's mirror does), or they are actually explicitly maintaining it. Either way, I'm sure it's good enough for our use.

comment:23 in reply to: ↑ 14 ; follow-up: Changed 5 years ago by vadz

Replying to bpetty:

The wx-commits-diffs is no longer seeing full source code diffs though. In the interest of sticking with GitHub maintained email hooks though, we might just retire wx-commits-diffs in favor of only offering wx-commits notifications.

I actually rather like getting diffs in the email, could we set up git-multimail on our mirror instead of relying on Github notifications?

comment:24 Changed 5 years ago by vadz

... third party libraries ...

I'm not sure about them. I'm not as anti-submodule as some other people and they do have advantages (e.g. not having to clone them at all if you don't need them, which is pretty common under Unix), but I'm not blind to their disadvantages neither. So I could live both with them and with Vaclav's approach.

One thing we need in any case is a clear (even if concise) guide to updating 3rd party libraries, especially if we don't use submodules (which are the standard way of doing this, like it or not).

comment:25 in reply to: ↑ 17 Changed 5 years ago by vadz

Replying to bpetty:

Additional files that still need updated:

Yes, definitely.

I think we can drop this one. With the switch to git, we can finally make build/tools/git-make-release the official script for making the releases.

And probably this one as well.

  • Buildbot updates.

Yes, at least build/buildbot/config/include/defs.xml.

comment:26 in reply to: ↑ 23 Changed 5 years ago by bpetty

Replying to vadz:

I actually rather like getting diffs in the email, could we set up git-multimail on our mirror instead of relying on Github notifications?

That certainly sounds much more useful. If you set this up Robin, can you edit the templates to include links to the GitHub changesets? Should be easy enough, and very helpful.

comment:27 follow-up: Changed 5 years ago by wbruhin

BTW, the comments on https://github.com/wxWidgets for wxPyWeb and wxPython repo's still mention that these are read only mirrors.

comment:28 in reply to: ↑ 27 Changed 5 years ago by bpetty

Replying to wbruhin:

BTW, the comments on https://github.com/wxWidgets for wxPyWeb and wxPython repo's still mention that these are read only mirrors.

The repo for wxPython actually still is a mirror. wxPython hasn't made this migration to git yet, just the Phoenix project and the main wxWidgets repo has.

comment:29 Changed 5 years ago by Bryan Petty <bryan@…>

In bdbadbeb53cb9458c10b5fa747f3275c1312eaec/git-wxWidgets:

Removed obsolete daily snapshot build scripts.

See #15588

comment:30 Changed 2 years ago by bpetty

  • Resolution set to fixed
  • Status changed from new to closed

Closing this ticket out with the final git submodule changes completed here:
https://groups.google.com/d/msg/wx-dev/qBIpwMa0ZyY/tTIUhO0cAQAJ

Note: See TracTickets for help on using tickets.