matplotlib-devel Mailing List for matplotlib

On Wed, Jul 21, 2010 at 6:38 AM, Andrew Straw <str...@as...> wrote:
> On 7/20/10 8:06 PM, Fernando Perez wrote:
>> On Tue, Jul 20, 2010 at 7:43 PM, John Hunter<jd...@gm...>  wrote:
>>
>>> The major issues I am aware of are:
>>>
>>> * what do to about all the various subdirs of the mpl trunk
>>> (trunk/toolkits/basemap, trunk/sample_data, etc..).  An svn commit to
>>> one tags all with a unique revision number.  In git, how do we
>>> synchronize between them?  Putting them all in the same tree would be
>>> monolithic and require huge checkouts.  Unlike svn, in git it is
>>> difficult/impossible to check out just a subdir (eg trunk/matplotlb)
>>> and also commit to it.  So we might end up having to informally
>>> synchronize parts of the trunk.  Eg, basemap rXXX requires mpl rYYY in
>>> the CHANGELOG or release notes.
>>>
>> Probably using a common tag across repos would be the easiest.  Any
>> time you want a known 'sync point', you tag all the relevant repos
>> with the same tag.  It then becomes very simple to write a little
>> script that will update checkout a bunch of repos sitting in the same
>> parent directory (each as its own dir, of course) at a common tag.
>> You can make up a convention for these special tags so that they are
>> always named with a given pattern (you could even use rNNNN if you
>> wanted).
>>
> We could also make a meta repository that uses git submodules (somewhat
> akin to svn externals). Each commit in a repo that links to submodules
> specifies a specific revision of the submodule repo, so this meta
> repository would be a fairly natural way of linking across several
> repositories at specific revisions. That being said, a convention to
> simply use the standard git tags would also work fine, and wouldn't
> require people to learn git submodules. So, it's really a question of
> sticking to a convention (that has a lesser learning curve) or using a
> new set of commands that would more or less do exactly what we want, but
> would have to be learned. I'm agnostic on the issue.

Is it possible to do both simultaneously for a while, to test whether
submodules works for us in practice? I have long been interested in
using something along the lines of submodules or nested repositories,
and a metarepository would seem to be exactly what we need. But this
business about cloning yielding empty submodules until you initialize
them, but then they are detached heads, remembering not using trailing
slashes when staging changes in a superproject... hopefully these will
not be issues, but we might need some time before we decide whether we
like submodules enough to embrace it long term.

I just finished a long run at work, and should have some time in the
next couple weeks during evenings and weekends that I could contribute
to the git and python3 transitions.

Darren

John Hunter <jdh2358@...> writes:

> * what do to about all the various subdirs of the mpl trunk
> (trunk/toolkits/basemap, trunk/sample_data, etc..).  An svn commit to
> one tags all with a unique revision number.  In git, how do we
> synchronize between them?  Putting them all in the same tree would be
> monolithic and require huge checkouts.  Unlike svn, in git it is
> difficult/impossible to check out just a subdir (eg trunk/matplotlb)
> and also commit to it.

I think git submodules could help here:

http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html
http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#submodules
http://github.com/guides/developing-with-submodules

The Basemap project would have a submodule pointing to some revision of the 
matplotlib project, etc.

Jouni

On 7/20/10 8:06 PM, Fernando Perez wrote:
> On Tue, Jul 20, 2010 at 7:43 PM, John Hunter<jd...@gm...>  wrote:
>    
>> The major issues I am aware of are:
>>
>> * what do to about all the various subdirs of the mpl trunk
>> (trunk/toolkits/basemap, trunk/sample_data, etc..).  An svn commit to
>> one tags all with a unique revision number.  In git, how do we
>> synchronize between them?  Putting them all in the same tree would be
>> monolithic and require huge checkouts.  Unlike svn, in git it is
>> difficult/impossible to check out just a subdir (eg trunk/matplotlb)
>> and also commit to it.  So we might end up having to informally
>> synchronize parts of the trunk.  Eg, basemap rXXX requires mpl rYYY in
>> the CHANGELOG or release notes.
>>      
> Probably using a common tag across repos would be the easiest.  Any
> time you want a known 'sync point', you tag all the relevant repos
> with the same tag.  It then becomes very simple to write a little
> script that will update checkout a bunch of repos sitting in the same
> parent directory (each as its own dir, of course) at a common tag.
> You can make up a convention for these special tags so that they are
> always named with a given pattern (you could even use rNNNN if you
> wanted).
>    
We could also make a meta repository that uses git submodules (somewhat 
akin to svn externals). Each commit in a repo that links to submodules 
specifies a specific revision of the submodule repo, so this meta 
repository would be a fairly natural way of linking across several 
repositories at specific revisions. That being said, a convention to 
simply use the standard git tags would also work fine, and wouldn't 
require people to learn git submodules. So, it's really a question of 
sticking to a convention (that has a lesser learning curve) or using a 
new set of commands that would more or less do exactly what we want, but 
would have to be learned. I'm agnostic on the issue.


>> * organizational stuff: how do we handle the notion of the central
>> repo?  Now that github support "organizations" this should be
>> relatively easy.  Andrew and I registered a matplotlib user acct at
>> github and created a gmail acct mplgithub as a central administrator
>> (mat...@gm... was taken, the bastards).  Email me offlist if
>> you are interested in obtaining the passwd for the github or gmail
>> admin accts -- but you should probably coordinate with Andrew who is
>> our point person as soon as he re-emerges.
>>      
> No need. Organizations let you designate more than one 'owner', so you
> can mark more than one person with full admin privileges without
> having to give out the password around.  I recently converted the
> extra ipython account to an organization, added Brian Granger as a
> second 'owner', and that's it.  You can then make as many teams and
> repos as you want within an organization.  The github org model is
> fairly simple but very effective (much nicer than how launchpad uses
> teams).
>    
I went ahead and switched our github.com/matplotlib account into an 
organization when github announced organization support a few weeks ago. 
Just now I added the jdh2358 and fperez users into the matplotlib 
owner's team.

(And I'm trying to re-emerge, I promise...)


>> * porting the buildbot to work w/ github commits
>>      
I should be able to handle that fairly easily. I do it for my other 
projects. (Bigger on my buildbot priority list is stopping the annoying 
occasional config directory multi-process conflict.


>> * related: porting the trunkdocs build to work with github commits
>>      
As this is run by the buildbot, it should be taken care of with the above.


For me, the main remaining issue is how do we want to pull the svn repo 
into git? Right now, the unofficial git repo at 
github.com/astraw/matplotlib is too big for my likes because it has a 
lot of stuff besides the matplotlib source code and its history. Before 
moving to an official git repository, I think we should prune down the 
main repository to just the source code files, their history, and no 
binary files. But then that leaves the question of what to do with the 
binary files.

-Andrew

On Tue, Jul 20, 2010 at 7:43 PM, John Hunter <jd...@gm...> wrote:
>
> The major issues I am aware of are:
>
> * what do to about all the various subdirs of the mpl trunk
> (trunk/toolkits/basemap, trunk/sample_data, etc..).  An svn commit to
> one tags all with a unique revision number.  In git, how do we
> synchronize between them?  Putting them all in the same tree would be
> monolithic and require huge checkouts.  Unlike svn, in git it is
> difficult/impossible to check out just a subdir (eg trunk/matplotlb)
> and also commit to it.  So we might end up having to informally
> synchronize parts of the trunk.  Eg, basemap rXXX requires mpl rYYY in
> the CHANGELOG or release notes.

Probably using a common tag across repos would be the easiest.  Any
time you want a known 'sync point', you tag all the relevant repos
with the same tag.  It then becomes very simple to write a little
script that will update checkout a bunch of repos sitting in the same
parent directory (each as its own dir, of course) at a common tag.
You can make up a convention for these special tags so that they are
always named with a given pattern (you could even use rNNNN if you
wanted).

> * organizational stuff: how do we handle the notion of the central
> repo?  Now that github support "organizations" this should be
> relatively easy.  Andrew and I registered a matplotlib user acct at
> github and created a gmail acct mplgithub as a central administrator
> (mat...@gm... was taken, the bastards).  Email me offlist if
> you are interested in obtaining the passwd for the github or gmail
> admin accts -- but you should probably coordinate with Andrew who is
> our point person as soon as he re-emerges.

No need. Organizations let you designate more than one 'owner', so you
can mark more than one person with full admin privileges without
having to give out the password around.  I recently converted the
extra ipython account to an organization, added Brian Granger as a
second 'owner', and that's it.  You can then make as many teams and
repos as you want within an organization.  The github org model is
fairly simple but very effective (much nicer than how launchpad uses
teams).

> * porting the buildbot to work w/ github commits
>
> * related: porting the trunkdocs build to work with github commits
>
> * how to handle the svn tree at sf -- should it mirror the new github
> tree or remain stale or simply removed?

I would freeze it during a transition period and later on make a
static backup of teh repo dump somewhere for historical purposes (and
just in case, disk is cheap).  I would then nuke it for simplicity of
administration, since on github people can still use svn if they want
to track a git repo:

http://github.com/blog/626-announcing-svn-support

I should note that I have not used this in practice, but a quick and
dirty test with the ipython repo seems to work (you just get the
master git branch though):

amirbar[junk]> svn checkout http://svn.github.com/ipython/ipython.git

[...]

amirbar[junk]> cd ipython.git/
amirbar[ipython.git]> svn info
Path: .
URL: http://svn.github.com/ipython/ipython.git
Repository Root: http://svn.github.com/ipython/ipython.git
Repository UUID: e94b1212-8258-e27c-589c-ce57b7db7bff
Revision: 2611
Node Kind: directory
Schedule: normal
Last Changed Author: fernando.perez
Last Changed Rev: 2611

> Please add to the list other issues that need to be handled.
>
> Of all these, I'm only concerned philosophically with the first.  The
> others are matters of time and work as people make the transition to
> the new server.  The first seems like a true potential workflow
> impediment for those who run off svn/git HEAD and analogues.

Others with more git expertise may suggest a different workflow, but
for that the tags approach, along with a couple of simple script
helpers to make creation/checkout of these tags a one-line operation,
seems like it should do the job.

Cheers,

f

> Yes, but there is a *lot* more involved in making the transition than
> just creating a git replica of the svn trunk.

Granted, and I'm probably forgetting many of them.  So it is probably
a good idea for us to enumerate them here as a checklist when we do
migrate.

The major issues I am aware of are:

* what do to about all the various subdirs of the mpl trunk
(trunk/toolkits/basemap, trunk/sample_data, etc..).  An svn commit to
one tags all with a unique revision number.  In git, how do we
synchronize between them?  Putting them all in the same tree would be
monolithic and require huge checkouts.  Unlike svn, in git it is
difficult/impossible to check out just a subdir (eg trunk/matplotlb)
and also commit to it.  So we might end up having to informally
synchronize parts of the trunk.  Eg, basemap rXXX requires mpl rYYY in
the CHANGELOG or release notes.

* we have to port the sample_data handling.  Jouni has shown that the
viewvc ETags in sf and github work the same way, so this should be a
minor hurdle.

* organizational stuff: how do we handle the notion of the central
repo?  Now that github support "organizations" this should be
relatively easy.  Andrew and I registered a matplotlib user acct at
github and created a gmail acct mplgithub as a central administrator
(mat...@gm... was taken, the bastards).  Email me offlist if
you are interested in obtaining the passwd for the github or gmail
admin accts -- but you should probably coordinate with Andrew who is
our point person as soon as he re-emerges.

* porting the buildbot to work w/ github commits

* related: porting the trunkdocs build to work with github commits

* how to handle the svn tree at sf -- should it mirror the new github
tree or remain stale or simply removed?

Please add to the list other issues that need to be handled.

Of all these, I'm only concerned philosophically with the first.  The
others are matters of time and work as people make the transition to
the new server.  The first seems like a true potential workflow
impediment for those who run off svn/git HEAD and analogues.

JDH