Integrating git with the Wolfram Language

John Fultz, Wolfram Research, Inc.
 — jfultz

Join the Conversation #WolframTechConf

What is git?

Sorry, this is the 60-second-intro-that-probably-still-leaves-you-lost version if you haven’t been exposed to it before...

Version control system

Decentralized

Represents commits, directories, and files relationally in a directed acyclic graph

Repos contain an object store, where the objects being stored are:

• Commits

• Trees

• Blobs

Object store is indexed by hexadecimal hashes known as SHAs

What is git — a visualization

SetDirectory[NotebookDirectory[]];
Manipulate[GitGraph[GitOpen["repos/TalkRepo.git"],selector],{selector,{"Commits","Trees","Blobs"}}]

Why access git from the Wolfram Language?

A natural extension of a symbolic language

Language support for tooling is a known good

Git is the leader in version control

Want to be able to support merging of notebooks

Turned out to be an invaluable internal tool

A basic introduction

Repo creation/acquisition

r=GitOpen["C:\\wolfram\\git\\GitLink"]

rclone = GitClone["repos/TalkRepo.git","../TalkRepo"]

rnew = GitInit["../NewRepo"]

A basic introduction

What can you do with a repo?

Get its properties:

GitProperties[rnew]

See its status:

GitStatus[rnew]

Add changes and commit changes:

Put[Sqrt[2],FileNameJoin[{rnew["WorkingDirectory"],"radical.wl"}]];

GitAdd[rnew,"radical.wl"]

c=GitCommit[rnew, "Added a radical"]

Get information on commits:

GitProperties[c]

Release resources:

GitClose[rnew]

Some more advanced concepts...

Accessing arbitrary revisions

ToGitObject["master",r]

GitRange[r,"master"]//Short

GitRange[r,"origin/feature/StandardNotebookMerge", Not["master"]]

GitGraph[%,"Commits"]

Some more advanced concepts...

Accessing trees and blobs

GitExpandTree[ToGitObject["master",r]]

GitReadBlob[%[[2,"Object"]]]

So what can we do with it?

Let’s do something practical...get some line counts

#Object&/@Select[GitExpandTree[ToGitObject["master",r]["Tree"],Infinity],StringMatchQ[#Name,__~~(".cpp"|".h")]&]

So what can we do with it?

Let’s do something practical...get some line counts

Clear[gitLineCount];
gitLineCount[commit_GitObject,pattern_]:=gitLineCount[Select[GitExpandTree[commit["Tree"],Infinity],StringMatchQ[#Name,pattern]&][[All,"Object"]]]/;GitType[commit]==="Commit"
gitLineCount[blobs:{___GitObject}]:=Total[gitLineCount/@blobs]
gitLineCount[blob_GitObject]:=(gitLineCount[blob]=Length[StringSplit[GitReadBlob[blob],"\n"]])/;GitType[blob]==="Blob"

gitLineCount[ToGitObject["HEAD",r],__~~(".cpp"|".h")]

Line counts

This is easy!

es=EventSeries[{#["Author"]["TimeStamp"], gitLineCount[#,__~~(".cpp"|".h")]}&/@GitRange[r,"master"]]

DateListPlot[es]

Line counts

Foiled by some messy data...what to do?

outliers=Select[GitRange[r,"master"],gitLineCount[#,__~~(".cpp"|".h")]>6000&]

Grid[{#Name, gitLineCount[#Object]}&/@Select[GitExpandTree[outliers[[1]],Infinity],StringMatchQ[#Name,__~~(".cpp"|".h")]&]]

Line counts

Final solution...clean up the outliers

es=EventSeries[{#["Author"]["TimeStamp"], gitLineCount[#,__~~(".cpp"|".h")]-gitLineCount[#,"mathlink.h"]-gitLineCount[#,"WolframLibrary.h"]}&/@GitRange[r,"master"]]

DateListPlot[es,Joined->False,Filling->Axis]

More fun with GitLink

A branch browser (how many languages can build this in 13 lines of code?)

icon=;
gitBrowse[tree_GitObject] := DynamicModule[{item},Row[{ListPicker[Dynamic[item],#Object->(#Name<>If[#Type==="Tree","»",""])&/@GitExpandTree[tree],Multiselection->False,ImageSize->{150,300}],Dynamic[gitBrowse[item]]}]] /; GitType[tree]==="Tree";
gitBrowse[blob_GitObject] :=
Button[icon,NotebookPut[GitReadBlob[blob,"NB"]],Method->"Queued"]/;GitType[blob]==="Blob";
gitBrowse[{}]:=""
gitBrowse[{item_}]:=gitBrowse[item]
gitBrowse[repo_GitRepo]:=
Deploy@DynamicModule[{branch=repo["HeadBranch"]},Column[{
PopupMenu[Dynamic[branch],repo["LocalBranches"]],
Dynamic@gitBrowse[ToGitObject[branch,repo]["Tree"]]}]]

gitBrowse[r]

[He’s showing off now...]

A branch browser with differencing

Code

Clear[objectMatchingName,colorizeLabel]
objectMatchingName[name_,list_]:=#Object&/@Select[#Name===name&][list]/.{x_GitObject}:>x
colorizeLabel[objs_List->label_]:=objs->Style[label,Switch[objs,
{x_GitObject,x_GitObject},Black,
{_GitObject,_GitObject},Orange,
{None,_GitObject},Darker[Green],
{_GitObject,None},Blue]];

Clear[diffBrowse,diffItemList]
diffItemList[{tree1_GitObject, tree2_GitObject}]:=
Module[{list1=GitExpandTree[tree1], list2=GitExpandTree[tree2],results},
results= Union[#Name&/@list1,#Name&/@list2];
results={objectMatchingName[#,list1],objectMatchingName[#,list2]}->#&/@results;
results= results /. {}->None;
colorizeLabel/@results];

diffBrowse[{tree1_GitObject, tree2_GitObject}]:=
DynamicModule[{item},Row[{ListPicker[Dynamic[item],diffItemList[{tree1, tree2}],Multiselection->False,ImageSize->{150,300}],Dynamic[diffBrowse[Flatten[item]]]},""]] /; GitType[tree1] === "Tree" && GitType[tree2] === "Tree" && tree1 =!= tree2;
diffBrowse[{blob1_GitObject, blob2_GitObject}]:=
DynamicModule[{item},Button[Row[{icon,Style["⇔",24],icon}],NotebookPut@NotebookTools`NotebookDiff[GitReadBlob[blob1,"NB"],GitReadBlob[blob2,"NB"]]]] /; GitType[blob1] === "Blob" && GitType[blob2] === "Blob" && blob1 =!= blob2;

diffBrowse[{___, obj_GitObject, ___}]:=gitBrowse[obj];

diffBrowse[{}] := "";

diffBrowse[repo_GitRepo]:=
Deploy@DynamicModule[{branch1,branch2},
Column[{
Row[{PopupMenu[Dynamic[branch1],repo["LocalBranches"],MenuStyle->{14,Bold,Blue}],
PopupMenu[Dynamic[branch2],repo["LocalBranches"],MenuStyle->{14,Bold,Darker[Green]}]},""],
Dynamic@diffBrowse[{ToGitObject[branch1,repo]["Tree"],ToGitObject[branch2,repo]["Tree"]}]
}]]

Demo

diffBrowse[r]

But it solved real problems for us

Post-processing repos migrated from cvs

Random git scriptability needs (e.g., let’s find all named and fully merged feature branches and clean them up)

Implementing a multi-repo management tool

Doing automated merge deployments of development branches (how we avoid having to have a “develop” branch slowing down our releases)

We built a thing...

Yummy, yummy dog food.  50+ developers use this every day at Wolfram.

And...

3-way notebook merging...

mergeRepo=GitClone["repos/MergeRepo.git","../MergeRepo"];
NotebookOpen[AbsoluteFileName["..\\MergeRepo\\Demo.nb"]];

GitCheckoutReference[mergeRepo,GitMergeBase[mergeRepo,"master","origin/feature_branch"]]

GitCheckoutReference[mergeRepo, "master"]

GitCheckoutReference[mergeRepo,"origin/feature_branch"]

GitCheckoutReference[mergeRepo,"master"];
GitMerge[mergeRepo,"origin/feature_branch"]

When’s it shipping?

A large amount is already done

We need to get some more coverage of git functionality

Some details may change before we ship.  I’ll update this talk to reflect any changes.

Update?  But where would we find it?

Github, of course!

https://github.com/WolframResearch

This is repo #1.  More to follow.