Technologie

Clone all repositories from a github organization

9 Comments

Tomorrow I will attend the first Hackergarten at Paris.
This is an event where active open-source committers can met potential contributors and help them to submit a (first) contribution.
I’ll be here to help those who would like to contribute on JenkinsCI and Apache Maven.
To prepare this event I wanted to prepare a checkout of all sources and a local repository with all required dependencies to avoid to see contributors losing their time by downloading the earth with Maven or checking out Subversion (and to not be blocked if we had some network issues).
For Maven which is hosted on Subversion it is long but easy because we have a special SVN directory using svn:externals to checkout all trunks in one step : https://svn.apache.org/repos/asf/maven/trunks/
But for Jenkins with several thousands of plugins and thus git repositories it is less easy to clone all of them.
I saw that there was everything needed in github APIs but it was late and I didn’t want to start to write a new script (One teacher told me a long long time ago that it was a quality to be idle – do it with less and less you’ll have to maintain).
I asked on jenkinsci IRC channel and thanks to Jørgen P. Tjernø I had my reply quickly.
With Ruby 1.8 (default version on Macos) :

sudo gem install json
curl -s https://api.github.com/orgs/jenkinsci/repos | ruby -rubygems -e 'require “json”; JSON.load(STDIN.read).each {|repo| %x[git clone #{repo["ssh_url"]} ]}'

With Ruby 1.9+, the json library is by default thus you just use :

curl -s https://api.github.com/orgs/jenkinsci/repos | ruby -rjson -e 'JSON.load(STDIN.read).each {|repo| %x[git clone #{repo["ssh_url"]} ]}'

With that I’m able to clone all repositories from jenkinsci github organization.
Tomorrow I’ll just have to launch a command like this to download all dependencies :

find . -name pom.xml -maxdepth 2 -exec mvn -f {} -Dmaven.repo.local=./maven-local-repo dependency:go-offline \;

Update : As there are more than 30 repositories in JenkinsCI Github organization, I had to call the clone command several times by adding ?page=X (X=2,..) after the url https://github.com/api/v2/json/repos/show/jenkinsci to grab all results pages.
Update 2 : We can also use per_page to increase the number of entries returned (max=100). Example (ruby 1.8) :

curl -s "https://github.com/api/v2/json/repos/show/jenkinsci?per_page=100&page=2" | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read)["repositories"].each {|repo| %x[git clone #{repo["url"]} ]}'

9 thoughts on “Clone all repositories from a github organization”

  2. thought i would just add, v2 api wasnt working for me

    the updated command for the v3 api would be (for ruby 1.8.7):

    curl -s https://api.github.com/orgs/jenkinsci/repos | ruby -rubygems -e ‘require “json”; JSON.load(STDIN.read).each {|repo| %x[git clone #{repo[“ssh_url”]} ]}’

    Note that i also use the ssh_url to clone the repo’s as the url now defaults to the api url which will not work and send these errors:

    fatal: https://api.github.com/repos///info/refs not found: did you run git update-server-info on the server?

  3. Nice,
    I was looking for this kind of tool, for a while.
    This is usefull to get all the GateIn source code, or all the ExoPlatform source code, to dig and search into it.
    Thanks a lot.

  5. Many thanks for the tip, very useful.

    I just had to change the “ssh_url” to “git_url” as I was getting public key auth errors (permission denied : publickey), I suppose the organization disabled public ssh cloning.

    Using “git_url” and Antoine H.’s command worked like a charm. Saved me a lot of time 🙂

    Thanks again

  6. Pingback: How to clone all repos at once from GitHub? - QuestionFocus

Comments are closed.