[robocup-legged] Legged Qualification FAQ

[William Uther]

Hi all,

   I've had a number of queries asking for further details on the  
qualification process.  I thought I'd post here to let everyone know  
what was going on.  I do NOT want to open a debate about the 2006  
qualification process - this is a zero sum game and I don't want to  
bias the decision towards the teams that complain loudest.  Non-zero  
sum suggestions are welcome.

A quick check reveals 35 teams submitted qualification material.  One  
thing that statistic doesn't show over previous years is the  
improvement in application quality: not only were there more  
applications this year, but they were generally of better quality  
than previous years.  That meant that it was harder to qualify this  
year than in previous years.

Some of the emails I have received have asked a few questions:

Q: How were the qualified teams decided?  Can I get a copy of the  
reviews?

A: The teams were decided by the technical committee.  Each committee  
member reviewed the materials submitted and then the teams were  
discussed over email.  Various committee members went back and  
reviewed specific aspects of specific teams where required, but we  
mostly did not generate specific review sheets.  There was no  
specific list of requirements by which the teams were sorted - the  
committee discussed various aspects of what makes a good team, but in  
the end each committee member gave their own weights to the various  
criteria.  These FAQs are my attempt to summarise the discussion.

Q: Can we increase the number of teams from 24?

A: Unfortunately not.  I would love to do this, as it would mean we  
wouldn't have to disappoint people.  Unfortunately we only have 4  
fields and a fixed time.  We simply couldn't fit the games.

   One possibility that has been floated is opening up the challenges  
to more teams.  This may well be possible, but we've only just  
started looking into it.  If teams that missed out on qualification  
are interested in this option, could they please email  
legged_tech at tzi.de

Q: Why are there so few new teams?

A: The technical committee has a slightly longer term view of 'new'.   
The first year a team is at the international competition, they  
generally learn a lot.  We would like them to get the opportunity to  
show that they can use this their second year.  From memory there  
were three teams in this category this year.  Adding these to the  
'new teams' count gives us four 'new teams'.  I think that is a  
reasonable number.

Q: Why are there 5 American teams and 5 Japanese teams?  Isn't that  
almost half the league from two countries.

A: It is even worse than you suggest!  There are 3 Australian teams,  
and that is a much higher proportion by population density.  The  
answer is that while we considered region when pre-qualifying teams,  
it was given a very low weight.  It was more used to break ties than  
anything else.  If there was no team from a region that might have  
carried more weight, but I don't think that was a major consideration.

   The combination of the last two points has affected some countries  
more than others.  Where the legged league has increased in  
popularity relatively quickly in one region of the world, it can be  
very difficult for new teams to secure a place in the international  
competition.  All I can suggest at this point is a) entering your  
local competition, and b) apply again next year.

Q: Code reuse: good or bad?

A: This has been a contentious issue for a while across many leagues  
in RoboCup, and there was much discussion at the RoboCup 05 legged  
league wrap-up meeting about code sharing as well.  Teams who had  
been working on their own code were concerned that a new team could  
come in, make a change in one area of the previous year's winning  
code, and do very well; whereas teams that wrote their own code were  
struggling to catch up.  Given that the new team is starting with  
very high quality code, their changes don't even necessarily improve  
the code.

 From the point of scientific development, code sharing is not a bad  
thing.  You want to stand on the shoulders of others, not their  
toes.  If you can modify someone else's code and win the competition,  
then that is wonderful.

 From the point of view of teams trying to acquire funding and  
reputation it is more of an issue.  As an extreme case, consider a  
team is simply a random change on the previous winner's code.   
They're not really helping scientific progress - they're just  
flooding the league with clones.  Other things being equal, a  
reduction in diversity is a bad thing.

Finding the right balance these two points of view is not easy - the  
fact that the TC would try to find a balance should not have been a  
surprise.

We are not the first RoboCup league to face this problem.  In the  
early days of the simulation league, CMU released their code.  The  
next year the league had many clones.  They also had a debate about  
how to deal with this.  CMU did not release their simulator team  
source code again (for a while - they may have started again - at  
least partially).  They decided that people could re-implement their  
published algorithms, but not just copy the code.

It should be noted that the NUbots, who placed 2nd in the  
international competition in 2005, completely re-implemented their  
code between the 2004 and the 2005 competitions.  This was done with  
a fairly small, but very skilled and motivated, team.  This suggests  
that re-implementing published algorithms is not too great a burden  
in the legged league.

We tried this year to at least get a look at how much code was  
changed from previous teams, rather than going on Technical Committee  
guesses.  Last year there were many clones.  This year we swung back  
the other way a bit.  I personally don't think we've found the  
perfect balance yet - but then I haven't seen anyone articulate the  
perfect balance, let alone try to implement it in the real world.

Q: How is qualification going to change in future years?

A: Each year the technical and organising committees decides who  
qualifies.  That means that the real answer depends on who is elected  
to the committees.  I can give my personal opinions though.

The quality of the league has been improving, and the number of teams  
applying has been increasing.  I see two options: increase the size  
of the international competition, or make the international  
competition more elite.

Increasing the size of the competition is going to cost money.  I  
don't know where we would get said money.

Making the international competition more elite is not necessarily a  
bad thing, but it does mean that the league as a whole would rely  
more on the local competitions - they could be used to decide more of  
the qualifications.  The problem with this is that the local  
competitions are often quite late - teams wouldn't know they were  
qualified until way after they should have booked their trip.

Suggestions for how the league should grow and change are always  
welcome.  I suggest they go to the robocup-legged at cc.gatech.edu  
mailing list (perhaps cc'd to the technical or organising committee).

Be well,

Will           :-}

--
Dr William Uther                           National ICT Australia
Phone: +61 2 8306 0424               Computer Science and Engineering
Email: william.uther at nicta.com.au      University of New South Wales
Email: willu at cse.unsw.edu.au                  Sydney, Australia

Web: http://www.cse.unsw.edu.au/~willu/   or  http://www.nicta.com.au/

NICTA email Disclaimer:
http://www.cse.unsw.edu.au/~willu/NICTAEmailDisclaimer.html
UNSW email Disclaimer:
http://www.eng.unsw.edu.au/emaildis.htm