Subsections

10. DARCS: The Revision Control System for SAGE

This chapter, which was written by Gonzalo Tornaria (and typeset by David Joyner), explains how to use the revision control system used by the SAGE developers. If you are a SAGE user not interested in becoming a SAGE developer then you will have no need to worry about this.

10.1 Getting started with DARCS

10.1.1 Resources

Homepage: <http://darcs.net/>

The ``official" manual: <http://darcs.net/manual/>

The wiki (``the community-maintained site for all things darcs"):
<http://darcs.net/DarcsWiki>

10.1.2 How to obtain DARCS

If you are debian based try:

        $ sudo apt-get install darcs
In general, look at this page <http://darcs.net/DarcsWiki/CategoryBinaries> for links to binaries and instructions for different systems (Windows, Mac OS X, several Linux, BSD, Solaris, AIX).

10.1.3 Do I have to install or configure ?

No need! If you have a ``darcs" binary that works, you are all set. Try darcs --version to see which version you have.

10.1.4 I need help

The man page is nice, but you don't really need it; darcs alone will give you a list of the available commands, and darcs COMMAND --help will give you detailed help on a particular command.

For more help, check the manual or the wiki.

10.2 DARCS and SAGE

10.2.1 Quick start

  1. Make sure you are upgraded to the latest SAGE:
            $ sage -upgrade
            [...]
    

  2. Then grab the repository from modular:
            $ cd $SAGE_ROOT/devel/
            $ darcs get http://modular.math.washington.edu/sage/dist/src/sage-darcs/
            Copying patch 65 of 65... done!
            Applying patch 65 of 65... done.
            Finished getting.
    

  3. Make this your SAGE code by symlinking the sage-darcs directory you just got to $SAGE_ROOT/devel/sage
            $ ln -snf sage-darcs sage
    

  4. And make the script sage-darcs/install executable:
            $ chmod +x sage-darcs/install
    

  5. Build and run this new version of SAGE:
            $ sage -br
            [...]
            sage:
    

  6. You are now running the development version of SAGE !!

10.2.2 Making changes

You can change whatever you want in this ``repo" you have now in

        $SAGE_ROOT/devel/sage-darcs
The revision control system will keep you safe from mistakes, as you can always undo any changes you make and go back to the original.

Moreover, if you record your changes as you work, you can selectively rollback, unrecord, or even get rid of some of your changes.

Let's try an example:

  1. chdir to your repo
            $ cd $SAGE_ROOT/devel/sage-darcs
    
    (from now on ``." will be that directory)

  2. Add
            print "hello world"
    
    at the end of
            ./sage/all.py
    

  3. See that DARCS noticed your change:
            $ darcs what -ls
            M ./sage/all.py +1
    

  4. Good, but we want the whole change:
            $ darcs what
            {
            hunk ./sage/all.py 232
            +print "hello world"
            }
    

  5. Fine, but we can't grok this (it doesn't show the context). What about a unified diff:
    $ darcs diff -u
    diff -rN -u -u old-sage-darcs/sage/all.py
    new-sage-darcs/sage/all.py
    --- old-sage-darcs/sage/all.py  2006-03-11 19:43:42.002107800 -0500
    +++ new-sage-darcs/sage/all.py  2006-03-11 19:43:42.290064024 -0500
    @@ -229,3 +229,4 @@
     from IPython.iplib import InteractiveShell
     InteractiveShell.exit = _quit_sage_
              
    +print "hello world"
    

    What is going on? Quoting from the manual:

    In a single DARCS repository you can think of two ``versions" of the source tree. They are called the working and pristine trees. Working is your normal source tree, with or without DARCS alongside. The only thing that makes it part of a DARCS repository is the _darcs directory in its root. Pristine is the recorded state of the source tree. The pristine tree is constructed from groups of changes, called patches (some other version control systems use the term changeset instead of patch).5.1 Darcs will create and store these patches based on the changes you make in working.

    What you have just changed is your ``working" tree. The ``pristine" tree has not changed, and the diff above is the difference going from ``pristine" to ``working".

    Keep in mind that the revision control is done on the ``pristine" tree, so if you do a zillion changes to your ``working" tree they will get all confused (you can still undo them one by one, but they are a zillion, and you won't remember which changes go together!).

    When you have a nice set of changes that you might want to preserve, you should ``record" them. This updates the ``pristine" tree with your changes, in the form of a new patch.

  6. Record the hello world patch:
        $ darcs record
        [...]
        What is your email address? Gonzalo Tornaria <tornaria@math.utexas.edu>
        hunk ./sage/all.py 232
        +print "hello world"
        Shall I record this patch? (1/?) [ynWsfqadjkc], or ? for help: y
        What is the patch name? test patch -- hello world
        Do you want to add a long comment? [yn] n
        Finished recording patch 'test patch -- hello world'
    
    (the email address is for authorship of the patch, in case you want to send it to someone else, and/or eventually ends up in the main SAGE repo - you should enter your email address, say the one you use to post to the mailing lists)

  7. Now ``working" and ``pristine" are equal again:
            $ darcs what -ls
            No changes!
    

  8. Let's see the change:
      $ darcs changes -p hello
      Sat Mar 11 20:01:46 EST 2006  Gonzalo Tornaria <tornaria@math.utexas.edu>
        * test patch -- hello world
    
    nice enough, there's only one patch with the word ``hello" :-)

  9. Mmm... what about other changes?
            $ darcs changes
            [... a LOT of changes! ...]
    

  10. Aha, let's see what's new since the last tagged version:
      $ darcs changes --from-tag sage
      Sat Mar 11 20:01:46 EST 2006  Gonzalo Tornaria <tornaria@math.utexas.edu>
        * test patch -- hello world
    
      Sat Mar 11 05:06:20 EST 2006  William Stein <wstein@gmail.com>
        * change factorial to use PARI C libraries "n!"; (40 times faster)
            
      Fri Mar 10 17:10:49 EST 2006  William Stein <wstein@math.washington.edu>
        * fix to matrices -- det only cached if immutable
            
      Fri Mar 10 01:24:37 EST 2006  William Stein <wstein@math.washington.edu>
        tagged sage-1.1.0
    

10.2.3 Compiling and running to test the patch

There are two commands related to building: ``sage -b" rebuilds this tree, while ``sage -br" rebuilds and runs the freshly built version.

        $ sage -br
        [...]

        hello world
        sage:
You can also run some tests as usual. When you make a simple change, it could be enough testing just that file, but when you make a more intrusive change you might want to run all the tests, etc.

10.2.4 Making a patch bundle to send patches

To interchange patches, rather than using a P2P network, DARCS lets you create a text file called a ``patch bundle", which contains patches together with some context information. The patch bundles can then be sent by whatever means (typically email, but you can print the bundle, use a pigeon carrier, and have the bundle OCR'd at the other end, cf. <http://www.faqs.org/rfcs/rfc2549.html> for some fun):

  $ darcs send -o hello.darcs
  Creating patch to "http://modular.math.washington.edu/sage/dist/src/sage-darcs/"...

  Sat Mar 11 20:01:46 EST 2006  Gonzalo Tornaria <tornaria@math.utexas.edu>
   * test patch -- hello world
  Shall I send this patch? (1/1) [ynWvpxqadjk], or ? for help: y
this creates a ``hello.darcs" text file with the patch bundle.

Notice that ``Creating patch to ..." remembers and uses the repo from where we pulled last time as a base. Indeed, when you choose a patch to send, DARCS will bundle all the depending patches that are not already in the base.

Your patch bundle will be good for anybody who is up-to-date with the same base as you; thus you should base your patch bundles in public repositories, and preferably the main SAGE repo, so that anybody can pull missing dependencies.

[aside:

If your system has a running sendmail configured to send mail to the outer world, you can send a patch directly from SAGE (if you don't, you might want to try ``msmtp", which is a pass-through from your system to some smtp server of your choice). If so:

   $ darcs send
   Creating patch to "http://modular.math.washington.edu/sage/dist/src/sage-darcs/"...

   Sat Mar 11 20:01:46 EST 2006  Gonzalo Tornaria <tornaria@math.utexas.edu>
     * test patch -- hello world
   Shall I send this patch? (1/1) [ynWvpxqadjk], or ? for help: y
   What is the target email address? wstein@math.washington.edu
   Successfully sent patch bundle to: wstein@math.washington.edu
there is no difference other than convenience in doing things this way instead of through a file.]

10.2.5 Forgetting a patch: unpull

Enough of SAGE greeting the world every time! Let's get rid of that patch:

   $ darcs unpull -p hello
   Sat Mar 11 20:01:46 EST 2006  Gonzalo Tornaria <tornaria@math.utexas.edu>
     * test patch -- hello world
   Shall I unpull this patch? (1/?) [ynWvpxqadjkc], or ? for help: y
   Finished unpulling.
Note that we used ``-p hello" to limit the possible patches. If you don't, you will be asked for all the patches; you skip out with ``d" (done).

Remember to ``sage -b" to rebuild SAGE.

Beware: unpull will eliminate the patch from both ``working" and ``pristine", meaning that the patch will be gone. This is completely safe with patches that you have pulled from a stable repo, because you can always pull them again. If you create a patch in your local repo, don't send it to anyone, and then unpull it, your patch is gone forever (actually, if you made a mistake, you might be able to recover from the file in _darcs/patches, which is not deleted, but this is tricky).

In our case, we saved a patch bundle, so that we can (and will) recover the patch from the patch bundle.

10.2.6 Receiving and applying a bundle

The recipient of the patch bundle can apply it with ``darcs apply"; since we unpulled the patch, we can play the role of recipient:

        $ darcs apply hello.darcs
        Finished applying...
Hint: ``darcs apply'' works as a filter, you can pipe directly from your mail program; but you won't be in the proper directory, so you will have to use the option --repodir=$SAGE_ROOT/devel/sage-darcs.

Remark: applying a patch bundle is ``safe" in the sense that darcs will refuse to apply a patch that would produce a conflict.

If you apply a patch bundle that someone sent you, and you decide you don't want it, you just ``unpull" it as above.

10.3 Upgrading and keeping up-to-date

10.3.1 Caution when upgrading with ``sage -upgrade"

The above works because we made a symlink to the working repo from $SAGE_ROOT/devel/sage, which is the default for building SAGE.

Be warned that if you upgrade with ``sage -upgrade" to a new version of SAGE, the symlink will be ``stolen", and (annoyingly) the repo will be moved to $SAGE_ROOT/devel/old/sage-darcs. You'll have to move your repo back and redo the symlink if you want to use that.

Not upgrading is not an option. Even if you stay up-to-date via the DARCS repo, eventually you will need to upgrade some other package.

As an alternative, here is what I normally do:

  1. download the packages that need to be upgraded:
            $ sage -update
            [...]
    

  2. check to see what is the version of the would-be-installed SAGE:
            $ cd $SAGE_ROOT/spkg/standard
            $ ls sage-*
            sage-1.1.0.spkg
    

  3. trick the package system to think this is already installed:
            $ cd $SAGE_ROOT/spkg/installed
            $ mv sage-* sage-1.1.0
    

  4. now build the /other/ packages that need to be upgraded
            $ sage -update-build
            [...]
    

10.3.2 Keeping the DARCS repo up-to-date

As SAGE development goes on, new patches will be added to the main repo. Presumably this will happen more often than releases, so that you can track development and be in the bleeding edge.

        $ darcs pull
        Pulling from "http://modular.math.washington.edu/sage/dist/src/sage-darcs/"...
        No remote changes to pull in!
Mmmm... we are already in the bleeding edge! Let's unpull a patch to fake that we are missing some patch:
        $ darcs unpull -p factorial
        [... interaction ...]
Now let's try pulling again:
        $ darcs pull
        Pulling from "http://modular.math.washington.edu/sage/dist/src/sage-darcs/"...

        Sat Mar 11 05:06:20 EST 2006  William Stein <wstein@math.washington.edu>
          * change factorial to use PARI C libraries "n!"; (40 times faster)
        Shall I pull this patch? (1/1) [ynWvpxqadjk], or ? for help: y
        Finished pulling and applying.
Notice that we were asked patch by patch (well, only 1 patch in this example) to see which ones I want to pull. we can pick any combination as long as dependencies are respected (but we don't have to worry about dependencies, as DARCS will force them on us).

If you see that there are a lot of patches, you don't care what they are and want to stop the interaction and pull them all, type ``a", or invoke pull with the ``-a" option as in ``darcs pull -a"

Remember that upgrading the SAGE library this way may require upgrading some of the other packages in the SAGE distribution. You definitely need to upgrade the SAGE distribution every once in a while (see discussion above).

WARNING: pulling changes can produce conflicts. Although this is perfectly safe for your recorded changes, this might be unsafe for unrecorded changes!

As a rule of thumb, you should never pull with unrecorded changes. You don't have to organize your changes before pulling, you might just record a temporary patch with something like

        $ darcs record -lam "TEMPORARY patch"
then you pull, and if you don't get any conflict you can
        $ darcs unrecord -p "TEMPORARY patch"
In case there are conflicts, you have the chance to try to solve them, but in the worst case (something goes wrong with the conflicts), you have a recorded change you can go back to!!!

10.3.3 What to do with conflicts

If you have some custom patches in your repo, when you pull from the main repo you may find some conflicts. This means that your patches modify the same line as some of the patches you are pulling from the main repo.

If a conflict occurs, DARCS explains that there was a conflict in certain files, and the relevant source files are *still* changed with text inserted pointed to the conflicts (look for v v v and ^ ^ ^). You then edit the source file to explain what you really want.

When you have a resolution for the conflict, you record a new patch; this new patch will depend on the two conflicting patches, and so long as the merge patch is pulled together with the conflict, no complain will raise.

Again: if two patches A and B have a conflict, and you pull both together, the conflict will be raised. You solve the conflict, and record a patch M which merges A and B.

Now say another person has a repo including patch A. If this person pulls patch B alone, there will be a conflict. But if patches B and M are pulled together, no conflict will arise!

10.4 Working with files

10.4.1 New files

If you add new files in your working repository, the commands won't see them by default. Some commands have a ``-l" option for this, e.g. (always in the repo dir)

        $ touch newfile
        $ darcs what -s
        No changes!
        $ darcs what -ls
        a ./hello.darcs
        a ./newfile
The same goes to ``darcs record":
        $ darcs record
        No changes!
        $ darcs record -l
        addfile ./hello.darcs
        Shall I record this patch? (1/?) [ynWsfqadjkc], or ? for help: q
        Record cancelled.
(I typed ``q" to quit)

Alternatively, you can be explicit about newfiles with:

        $ darcs add newfile
        $ darcs what -s
        A ./newfile
        $ darcs what -ls
        A ./newfile
        a ./hello.darcs
(note A versus a)

Warning: this doesn't add the file to the ``pristine" repository, it just schedules for addition, this will need to be recorded as any other patch!)

10.4.2 Removing files

We can explicitly remove files from revision control:

        $ darcs remove newfile
        $ ls newfile
        newfile
note that this didn't delete the actual file, only removes it from control version tracking. In fact:
Remove should be called when you want to remove a file from your project, but don't actually want to delete the file. Otherwise just delete the file or directory, and DARCS will notice that it has been removed. Be aware that the file WILL be deleted from any other copy of the repo to which you later apply the patch.
(from ``darcs remove -help'')

Indeed:

        $ rm -f new
        $ darcs what -s
        R ./new
[ As with ``add", this doesn't actually change ``pristine", it just schedules the removal (in this case, though, the scheduled addition is cancelled with the scheduled removal, since we didn't record anything). ]

Life saving device:

        $ darcs revert
        hunk ./new 1
        -darcs whatsnew   |less
        Shall I revert this patch? (1/?) [ynWsfqadjkc], or ? for help: y
        Do you really want to revert these changes? y
        Finished reverting.
        $ ls new
        new
        $ darcs what -s
        No changes!

10.4.3 Renaming (moving) files

Unlike remove, this needs to be done with DARCS, so that revision control knows the file was renamed (this will allow DARCS to commute patches past the rename!)

        $ darcs mv notes.txt notes.text
        $ ls notes.*
        notes.text
        $ darcs what -s
         ./notes.txt -> ./notes.text
        $ darcs record -am "moving notes.txt to notes.text"
        Finished recording patch 'moving notes.txt to notes.text'
        echo add some text >> notes.text 
        $ darcs what -s
        M ./notes.text +1
        $ darcs record -am "add some text to notes.text"
        Finished recording patch 'add some text to notes.text'
Let's create a patch for the added text (but NOT for the rename, uh)
   $ darcs send -o addtext.darcs -p add   
   Creating patch to
   "http://modular.math.washington.edu/sage/dist/src/sage-darcs/"...

   Sat Mar 11 22:27:40 EST 2006  Gonzalo Tornaria <tornaria@math.utexas.edu>
     * add some text to notes.text
   Shall I send this patch? (1/?) [ynWvpxqadjkc], or ? for help: y
Now let's forget the last two patches
        $ darcs unpull -p notes.text
        [... interaction ...]
        $ ls notes.*
        notes.txt
        $ cat notes.txt
        darcs diff --from-patch "mpz_pylong: add notices"
So, the file is back to its place, our patch is gone, can we apply the patch bundle? Yes!
        $ darcs apply addtext.darcs
        Finished applying...
        $ cat notes.txt
        darcs diff --from-patch "mpz_pylong: add notices"
        add some text
Note that the patch was automatically commuted with the rename.

10.5 Tips and tricks

10.5.1 DARCS configuration

There is a configuration directory for DARCS, where you can override some defaults. In linux this is in ~/.darcs. Some examples:

10.5.2 Forcing ``darcs apply" even with conflicts

Use the ``--mark-conflicts'' option for this. As above, you should NOT have any unrecorded changes in your repo. You then resolve the conflicts as above.

Alternatively, if you are the main repo maintainer, you can bounce the patch bundle to its author, and ask him to pull the main repo and merge the patch. It should be easier for the patch author to understand how to fix the conflicts.

10.6 Other useful commands to try

  1. darcs revert

    this reverts ``working" to the state of ``pristine" (i.e. to the last recorded version of the tree).

  2. darcs unrevert

    you have *one* chance to undo a revert, but if you pull, unpull, apply, record, unrecord, etc., your chance is gone.

  3. darcs unrecord

    this is the inverse of record. Like unpull, it takes a patch out of your ``pristine" tree, but you KEEP the changes in your ``working" tree. Presumably you will change things a bit and record another patch.

  4. darcs amend-record

    this is like doing ``darcs unrecord" followed by ``darcs record". You don't get a chance to change the patch name, description, or dependencies, but the timestamp is changed (so the you obtain a new patch which has nothing to do with the old one).

  5. darcs rollback

    this applies an ``UNDO" patch to your ``pristine" tree, but keeps the changes in your ``working" tree. In a sense, this is the same as ``darcs unrecord", but the patch stays in the repo, so you can go back to the patch some other time...

  6. darcs resolve

    if you messed up while resolving a conflict, this will mark the conflicts again. BEWARE: you will loose all unrecorded changes!!

10.6.1 Other things to try

  1. copy your the whole directory of your repo into a webpage (_darcs included). This makes you a public repository and others can pull patches from there! (as easy as 1-2-3)

  2. try to use the ``test" feature of DARCS with SAGE (tricky)

10.7 Important note on changing patches

Quote from the manual:

Keep in mind that the patches are your history, so deleting them with unrecord makes it impossible to track what changes you really made. Redoing the patches is how you ``cover the tracks''. On the other hand, it can be a very convenient way to manage and organize changes while you try them out in your private repository. When all is ready for shipping, the changes can be reorganized in what seems as useful and impressive patches. Use it with care.

All patches are global, so don't ever replace an already ``shipped'' patch in this way! If an erroneous patch is deleted and replaced with a better one, you have to replace it in all repositories that have a copy of it. This may not be feasible, unless it's all private repositories. If other developers have already made patches or tags in their repositories that depend on the old patch, things will get complicated.

In short: once you start distributing a patch, you should not change it anymore. Instead of using amend-record, you should record a NEW patch that amends the old one. Otherwise, you'll start to have headaches. For instance, if someone bases some patches in your old patch, and you base some patches in your new patch, you'll have a hard time merging...

In particular, once a patch goes into the main repo, it should never go out for any reason.

A developer can have several trees to avoid problems with this, e.g.:

[ It might be possible to unify the last two if testing for SAGE can be implemented. Indeed, it is possible to have DARCS run a test before approving a patch that is applied with ``darcs apply". If the test fails, the patch bundle is bounced together with the output of the test.

This would be really nice; one could send (signed) patches by email to ``darcs apply" (e.g via procmail), and have them be tested in the background, and only applied if all tests pass... that's peace of mind :-)

The difficulty is that SAGE is not self-contained; there isn't right now an easy way to build/run SAGE from a temporary directory without touching the whole SAGE environment.

IF we had some way to ``build and run, and run the tests" from an alternate directory, and without installing anything to $SAGE_ROOT, this would be possible (and very cool). ]

See About this document... for information on suggesting changes.