Undergraduate Capstone Open Source Projects

Archive for the ‘Mercurial’ Category

Mercurial: the end of the beginning

Posted by Anton Markov on 2010/04/05

Today marks the last day of classes at the University of Waterloo, and so ends the official portion of the UCOSP project for Tessa, Wendy, and myself. During our last meeting on April 1st, we tied some loose ends on our patches and discussed our experiences in the project. I will save the details for the final reports, but we all agreed that we’d like to continue contributing to Mercurial as time permits.

Paul and Alex have a little more time before official end of classes at UBC, and we’ll probably be hearing from them after the meeting this week.

Status updates:

  • Bfiles auto-* integration
    • auto-status and auto-update have been accepted into bfiles
    • auto-put and auto-refresh are almost there
  • Wendy and Tessa continued to improve on their warm-up projects:
  • Paul has a working version of bfilesify which converts existing mercurial repositories to use bfiles. We are playing around with it to provide feedback.
  • Alex has been working on his ever-popular case guard extension.
  • I have started work on implementing my extended status proposal which aims to create closer integration between bfiles, subrepos, and core Mercurial code.

As always, you can find detailed status reports on our Wiki at https://ucosp.fogbugz.com/default.asp?W44

Posted in Mercurial, Status | Leave a Comment »

The Real Challenge of UCOSP

Posted by Anton Markov on 2010/04/05

“The real challenge of GSoC is not the coding, it’s learning to work with a community,” wrote Matt Mackall to the Mercurial mailing list a couple days ago with regard to the upcoming Google Summer of Code projects: http://selenic.com/pipermail/mercurial/2010-April/030996.html

I believe this applies equally well to us: the real challenge of UCOSP is not the coding, it’s learning to work with a community. This is especially true for the Mercurial team because we are working with such a large, established, global community of developers. Working within our UCOSP team was alright for our initial warm-up projects where we built new Mercurial extensions or added isolated features, but it became very important when we started working on the bfiles extension. We collaborated extensively with Greg Ward, the creator of bfiles. Towards the end, he helped us extensively with combining and polishing our patches, and as a result, they integrated smoothly into his tree. The more we engaged the developer community, the more productive we became.

The post that I link to has some other great pointers for anyone new to an opensource project – not necessarily those doing it for credit or money. It’s worth a read.

Posted in Mercurial | 1 Comment »

mercurial # the end if nigh

Posted by alexandru on 2010/03/26

For some of us on the Mercurial crew next week is the closing chapter of the UCOSP adventure. Tessa, Wendy and Anton from Waterloo will be finishing their term, and with that their contributions to Mercurial as part of the program. That’s not to say they’ll be done with Mercurial for good—the general consensus is that we all had a great time. Paul and myself still have until mid-April to get the remainder of our work submitted.

For the last couple of weeks we’ve all been focusing on bfiles, the big/binary file extension for Mercurial. For the most part, the functionality that’s been added is to automate certain functions so that the user doesn’t have to remember to do so:

Tessa finished up autoput, including tests for the feature. She’s also submitted a few more changes for her globexclude patch.

Wendy worked on autorefresh and autoupdate for bfiles, as well as some final tweaks for her tagmerge extension.

Anton polished up autostatus for bfiles and began work on extended status support.

Paul has been working on bfilesify, an extension to convert a regular Mercurial repository to a bfiles one. This means all large binary files get detected and plugged into bfiles, with the history accurately maintained.

Alexandru worked on supporting HTTP PUT for bfiles, so that users can push their large files to remote repositories via HTTP (currently, only SSH is supported). Additional work went into caseguard to provide a feature that allows users to check an existing repository for filename issues. Alexandru also apologizes for talking in the 3rd person for this section, it was done in the interest of post formatting.

More information available from our weekly Wiki status page and the IRC log.

Posted in Mercurial, Status | Leave a Comment »

Mercurial is still flowing

Posted by Anton Markov on 2010/03/05

In Mercurial land, we’ve made progress in closer integration between bfiles and core Mercurial commands, a tool to “bfilesify” a Mercurial repository, and further work on the caseguard extension. More details below.

Due to various scheduling conflicts, we did not have a status meeting last week. This week the meeting happened on Wednesday at 9pm EST. We’ll be back to Thursdays at 6:30pm EST starting next week.

Chat transcript: https://ucosp.fogbugz.com/default.asp?W38

Status reports for two weeks:

Closer bfiles integration: as I’ve discussed in a previous post, bfiles is an extension to store history of big and binary files outside the repository. Currently ther bfiles extension has its own set of commands: ‘hg bfstatus’ analogous to ‘hg status’, ‘hg bfadd’ analogous to ‘hg add’, etc. We want to integrate them into the core commands like ‘hg status’.

While I spent my time discussing auto-status integration and debating the proper way to do it, Tessa and Wendy made significant progress on auto-put, auto-update, and auto-refresh. We are ready to try using and testing the parts together before making them ready for submission (as individual patches/branches).
[Edit 3/5/2010: removed confusing wording about “combining into a single patch”]

Paul has taken up the torch on a much-needed extension we call ‘bfilesify’ which would take a regular repository and convert it to using bfiles where appropriate. This is an exciting project which should greatly ease the transition to bfiles.

Last but not least, Alex has been polishing up caseguard and improved its performance to the point where the impact on speed from using caseguard is negligible. Caseguard has also become a somewhat regular topic on the mercurial mailing lists – a good indication that it sees real use.

Posted in Mercurial, Status | Leave a Comment »

Wendy’s Midterm report

Posted by q6yang on 2010/03/01

It has been a very good learning experience for me to work with Mercurial team so far.

What I learned:

To be honest, I had no experience in python coding or Mercurial at the beginning of the semester.  I was terrified that I had no idea how to start my worm-up project tagmerge extension at the code sprint. Thankfully, Benjamin was really patient with me and guided me where to start off.  After couple of weeks working on tagmerge, I become comfortable with python and familiar with some mercurial libraries.

Same as many other members on the team, this is also my first experience working on an open-source project. In my co-op experience, I usually only discuss issues with my supervisor or co-workers.  Issues are only discovered during the QA process. When doing development in Mercurial, we have to handle both development and do our best to cover all the test cases. It is very important to discuss issues we encountered through mailing list or IRC instead of just with only supervisor or co-workers, because other developers may identify scenarios I missed or give me ideas on how to implement some features.  All these suggestions are very helpful for my work and my own development.

What I work on:

We each started with a warm-up project to get familiar with Mercurial. I worked on tagmerge extension to merge hg tags. I successfully finished the development and testing and announced it a few weeks ago. However, I still need to work on some improvement before get it published.

The team is currently focused on bfiles extension now. Three of us from UW are working on integration of some bfiles commands. As everyone else mentioned, we are a bit slow due to the midterm period. I had my part of design set, and hopefully, will finish the draft by this week.

Overall, I think I’m doing good so far. I’ve learned a lot from my projects, keeped up with weekly reports, meetings and tried to join discussions on mailing lists. However, I always started real working from Monday, which leaves me less time to get reviews of my code before the Thursday meeting. For the rest of the semester, I will improve my time management and try to contribute more of my work to Mercurial.  I’m excited about the challenges ahead.

Posted in Mercurial, Status | 1 Comment »

The Mercurial Team’s Midway Report

Posted by Benjamin on 2010/02/28

When we set out at the beginning of UCOSP, my goal was for us to contribute significantly to Mercurial by the end of the semester. How well have we accomplished that goal so far?

At the beginning of the semester, all of the students were given small, self-contained bug fixes and features that we’d discovered people wanted over the course of developing Kiln, mostly in the form of Mercurial extensions. Four of these were quickly completed: cedit, a command-line-based way to configure Mercurial; cache-annotations, which makes annotating files repeatedly ridiculously fast; globexclude, which allows you to exclude files based on their glob pattern when doing conversions to Mercurial; and caseguard, which warns you when you are committing files that have names that will prevent Mac and Windows users from using your repository. Of these, globexclude and caseguard will be submitted (with some slight modifications) for inclusion into Mercurial 1.6, and cache-annotations already ships with every version of Kiln. This part of the project I consider a complete success.

Since then, we have transitioned from everyone doing a little project to everyone working on bfiles, an extension to Mercurial that is designed to ameliorate using Mercurial with projects that involve large binary files.

I would be lying if I said I wasn’t slightly disappointed with the work done on bfiles so far—not really through any fault of my students, as much as due to the nature of what they’re trying to accomplish. bfiles is necessarily a more complex problem than the initial, self-contained feature sprints: bfiles requires some changes to Mercurial’s workflow, and these changes don’t always have a single, straightforward solution, and so everyone has to take the discussion to the Mercurial mailing list to figure out how to proceed. Having these discussions is unquestionably not a waste of time. These are delicate issues, and working with the Mercurial community to determine how to proceed is an absolutely necessity. This has, unfortunately, had the side-effect of delaying writing some of the code we wanted for bfiles until we can get some of the design straightened out.

I’m nevertheless still very optimistic for the second half of the semester. We’ve sorted out a lot of the hard issues, allowing everyone to begin focusing on writing code again (even if only for initial prototypes, in some cases). Simultaneously, we’ve begun working on some aspects of bfiles that do not require so much debate to ensure that we’re doing the “right” thing. This means we can continue to code, while still carefully consulting the community on ambiguous areas.

Balancing requirements gathering with simply getting the code done can be a complicated affair on open-source and commercial projects alike. I think we’ve been erring too much towards discussion in the last couple of weeks, but as long as we can correct ourselves soon and get focused on the code again, I’m still confident that we’ll get nearly everything accomplished this semester that we set out to do.

Posted in Mercurial | 1 Comment »

alexandru’s Half-Time View

Posted by alexandru on 2010/02/26


I’ve used open-source software for the last 7 years (which is feels like a blip in Internet time). I was aware since the beginning that these projects thrive on contributions from people all over the world. And I knew that these tools had bugs and had roadmaps with desired features waiting on some developer to tackle. But before UCOSP I’ve never made a solid contribution to anything. I always imagined expert programmers are the only ones that have a business getting involved in these projects and submitting patches and extensions. So I stayed on the sidelines for the most part, always saying I’ll do it later, when I got better at programming.

When UCOSP started, I was thrilled to be working on the Mercurial project. I was also thrilled to finally have the time to learn Python, and terrified by not really knowing any. I had my doubts, early on, about being able to get something decent done by the end of April. This wasn’t the first time I dived head-first into something of the sort (having completed 5 internships at various software outfits during my student career), but it felt different. It was going to be out in the world for people to use. I was finally working on an open source project, planning on contributing something I knew was needed.

Guess what? It’s going great! Over the past years I kept hearing that failing is not a bad thing, that if you don’t get it right the first time the World won’t implode. I’ve learnt that failure comes more often from the desire to be perfect than from the work itself, because perfection is the enemy of productivity, and over-engineering is something we’re all very prone too. So I made bite-sized goals and I’ve incrementally built caseguard, my Mercurial extension that’s meant to help developers avoid filename collisions (on case-insensitive filesystems) and the feedback so far has been good. The Mercurial devel mailing list has been invaluable for both feature recommendations and help with the existing design, and the UCOSP mentor, Benjamin Pollack has been similarly patient with us.


Acquaintance to how open source works is an indispensable experience, especially for students, because it’s so different from anything else we’re normally exposed to. All great software, whether open source or not, is built because someone needs it. Often times students and junior developers don’t get an opportunity to pursue their ideas inside a project because the requirements are either filling up the entire course time, or the senior developers have already outlined the things that need to get done. But with open source (and, by extension, UCOSP) we get to own our code and take it in whichever direction we feel it should go. Community feedback can be a driving force, but it’s not law. Learning how to prioritize, which features are important and which nice-to-have ones you should delay until a time they’re actually needed, quieting the inner voice that wants to keep adding things because some believe that more is better, these are all skills that matter greatly in software development. Sometimes, they might decide a project’s success more than the pure programming prowess of the developers involved.

I wish I started contributing to open source earlier, but it’s never too late. The biggest failure comes from not trying at all; anything else is already better. My experience in UCOSP so far proves my point, and I hope that’s true for everybody across all teams this term. I hope UCOSP-like courses become the norm in Computer Science/Engineering programs, because they help a developer’s education once all the theory has been absorbed throughout the program years.

second half

For the second part of the term I’m going to continue work on caseguard, targeting inclusion in the Mercurial distribution for 1.6 (targeted release July 1st). It may be an extension included in the core distribution or it may be part of the actual tool itself (over IRC, the Mercurial project lead Matt Mackall suggested I submit a patch to see how it would work in core) and that’s all very exciting. Python is a very elegant language and I’m liking it the more I learn it, and I’m lucky to be working with a code-base that many consider to be an example in programming practices (by and large, but not entirely). I’m also working with everyone else on improving bfiles, an extension meant to improve the way Mercurial deals with big/binary files, and my first order of business is supporting HTTP uploads.

Posted in Mercurial, Status | 3 Comments »

Anton’s Midway Update

Posted by Anton Markov on 2010/02/24

It’s hard to believe that we are already 1/2 way through the UCOSP Mercurial project. It has been an amazing learning experience from both a technical and a social point of view.

I started out this project with some limited experience of using Mercurial on my own system. Not only did I have to learn Mercurial in the context of a team, but I also had to put myself in the shoes of users very different from myself. We are working on features for large teams with large repositories who have different requirements from individual developers. I will get back to this later.

My experience with Python was equally limited. I had used it for a couple of school assignments in the past, but as most of you know, one-time throw-away assignment code is nothing like a mature open-source project. As the Python coding guidelines quote, “Code is meant to be read by humans and only occasionally by computers.” Python is a powerful language, and I’ve been blown away by how clear and concise the code is. On the other hand, like any powerful tool, it must be used wisely and consistently.

From a social point of view, this is my first experience contributing code to an open-source project. The basic concept of bug tracking, specifications, and code reviews were familiar to me from co-op terms, but they take on a new light in an open-source project. While there are three of us working on Mercurial from the University of Waterloo, and we have met to discuss our work face-to-face, we quickly learned that any important discussion must take place on the public mercurial-devel mailing list. It is tempting to talk and decide things amongst ourselves, but it is important to involve the other developers and users of Mercurial. This is especially important when modifying other people’s code or working with scenarios that we don’t have personal experience with.

We started off the semester poking around and learning about Mercurial. The real work started at the UCOSP code sprint in Toronto where we spent 3 days hacking away on Mercurial together, in the same room. We all had small warm-up projects to work on. I worked on an extension to cache the results of the ‘hg annotate’ command which shows, for each line of a file, the last time when it was modified. This project went well and the code was polished up in another week. I’ve since released it publicly on Bitbucket, and it has been integrated into FogCreek’s Kiln product.

Our next challenge was the bfiles extension. The bfiles extension is designed to make working with large binary files more convenient. Instead of storing the entire history of every file in every clone of the repository, the history is stored on a central server and each revision is downloaded as needed.

We seemed to hit a slow period with bfiles. It is a large extension which messes with various internal Mercurial functions. Furthermore, our tasks were very broadly-defined. We spent three weeks discussing different approaches on the mercurial-devel mailing list and getting the functional specs right. Finally I was able to start coding over reading week, but I hit implementation decisions and went back to the mailing list to discuss them.

Greg Wilson has taken every opportunity to emphasize the importance of communication on this project. I couldn’t agree more. Writing this update has helped me to organize my thoughts. I am off to do work.


Posted in Mercurial, Status | 1 Comment »

Mercurial Weekly Report (Week 8)

Posted by q6yang on 2010/02/19

We continued our work on bfiles and improving our patches this week.


  • cedit (paulitex):
    • Submit unit test and rege fix for iniparse
    • Documentation for credit
  • tagmerge(Wendy):
    • Use tag.py functions to read tags
    • Changed the merge algorithm
    • Debugging writing tags problem
  • convert (Tessa):
    • Tested standard mercurial matcher
    • Refactored filemapper to use the mercurial match
  • caseguard (alexandru):
    • Removed the protection for `hg rm’
    • Code clean-up/polish


  • auto-status(antmar): implemented and submitted the initial version (wraps “hg status” command)
  • auto-put(Tessa): read through more code about it and revisited the proposal
  • auto-refresh & auto-update (Wendy): created detailed proposal on the mailing list and discussed about the proposal
  • bfiles HTTP upload(alexandru): started investigate this issue

We further discussed the problems we met this week and plan to finish the patches and submit a working code draft for bfiles command integration.

Details are in IRC meeting and status report .

Posted in Mercurial, Status | Leave a Comment »

Mercurial Weekly Status Report (Week 7)

Posted by tessastarkey on 2010/02/12

Here is a summary of what we’ve been up to this week. As usual, the IRC transcript of the meeting and our more detailed   status report are posted.

  1. Continued work on our our patches  that were submitted last weeks
  2. Progress with bfiles:
  • Benjamin set up a Linode server for us to develop and test the network-related  bfiles issues
  • Continued  discussion on the  the mercurial-devel mailinglist with seasoned mercurial developers regarding our ideas for autorefresh  and how to keep the bfiles in the respository in a consistant state

After we finished discussing UCOSP-specific issues, we move to the main Mercurial IRC channel where we further discussed getting  rid of bfrefresh, and having the refresh happen automatically.

Posted in Mercurial, Status | Leave a Comment »