Caching build results

This is a “technology/concept identification” question.

I have a rather large project in a DCVS repository.

Build times are loooong (about an hour) from a full clean.

The project is already divided into subprojects, with incremental building. But it still takes a very a long time when switching branches, etc., since I’ll often have to compile a lot.

I had an idea of caching the subproject outputs (I’m using Scala, so in this case, it’s a bunch of .class files), and publishing them with the hash of the source as the key. Before a build, I would download any outputs that matches the hashes of any of the subprojects. The “pain” of building would be shared and reused across the organization, including the CI server.

I can’t have been the first person to think about this.

Is there a name for this caching-by-source (rather than explicit version)? Is it commonly implemented? Any existing tools around this?


EDIT: I have found one example in scons, a tool used primarily for C/C++ builds.

3

sbt

The standard build tool for scala is sbt (Scala Build Tool). Within sbt there are three ways to do dependencies:

  • Declarations in your project
  • Maven POM files
  • Ivy configuration

To use Maven, one would need to specify the resolvers for the location of your repository:

resolvers += "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository"
resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"

You could also override this by creating a local configuration file: ~/.sbt/repositories

[repositories]
local
my-maven-repo: https://example.org/repo
my-ivy-repo: https://example.org/ivy-repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]

If you really had to, you could also specify a url and set up some way to host it locally on a server you control. Note that this isn’t an ideal solution

libraryDependencies += "slinky" % "slinky" % "2.1" from "https://slinky2.googlecode.com/svn/artifacts/2.1/slinky.jar"

However, this is only half of the battle. Sure you can pull the jars from some place, but you also need to put them there. If you are not putting the modules out there for someone else to download and use, they are going to have to recompile them.

You could set up a Sonotype nexus server and then publish to it:

publishTo := Some("Sonatype Snapshots Nexus" at "https://oss.sonatype.org/content/repositories/snapshots")

Or you could have a commonly mounted repository (I don’t think this would be a best solution):

publishTo := Some(Resolver.file("file",  new File( "path/to/my/maven-repo/releases" )) )

You can configure it to publish some things here and some things there:

publishTo := {
  val nexus = "https://oss.sonatype.org/"
  if (isSnapshot.value)
    Some("snapshots" at nexus + "content/repositories/snapshots") 
  else
    Some("releases"  at nexus + "service/local/staging/deploy/maven2")
}

But the thing there is that you need to publish it in order for other people to download it.

maven

Maven is (or should be) well known to Java developers. It isn’t something that is commonly used with scala. But, it is possible to get scala and maven to work together. Along with src/main/java you will have src/main/scala. You will also have a plugin in your .pom:

<plugin>
  <groupId>net.alchim31.maven</groupId>
  <artifactId>scala-maven-plugin</artifactId>
  <version>3.1.3</version>
  <executions>
    <execution>
      <goals>
        <goal>compile</goal>
        <goal>testCompile</goal>
      </goals>
    </execution>
  </executions>
</plugin>

At this point, you are using maven and you are deploying maven artifacts to the Sonotype nexus server.

general things

All of this, with either approach depends on something else though. You need to have your project modularized so that small jars can be built, compiled, and deployed to the artifact server.

If your project is not configured this way, you will need to change it so that small independent parts are distinct.

You may find that the easiest thing to do (so you don’t have to remember to do a publish or deploy) is to set up a continuous integration server that is fetching changes from your version control server, building them, and automatically deploying successful builds to the server.

This also implies some degree of discipline with versioning of the artifacts. You may wish to look at semantic versioning. The identifier for the release info so that you don’t step on each other’s toes and deploy competing snapshots or overwrite someone else’s version number. You will find yourself with versions like 1.0.3-pauld.0.1 so that they are namespaced.

And thus, you will specify with sbt or maven that this project uses some artifact at some version. That pre-release version number will likely have the branch or author name in it. It will be deployed to a server where you can then pull it. When someone else switches to that branch those modules will be available for the build tool to download and use without having to recompile them.

languages and tools

A key thing to remember here is that one doesn’t deploy .class files. Instead, you are deploying .jars. Everything is built around either locally compiled .class or downloaded libraries. While one could conceive of doing this with other languages, with Java and Scala the IDEs would get very annoyed if the .class files and the .java or .scala files became out of sync. If you’re using vim and javac or scalac, then you might find other tools that pull singly compiled assets useful.

However, I’m going to point out that this isn’t as trivial as it would first appear. You will find things like Outer$Inner.class and Outer$1$3.class in instances where you’ve got inner classes (the second example is the compiled file for the third anonymous inner class in the first anonymous inner class in the class Outer). With anonymous inner classes, this is position dependent in the file – add some code and Outer$1.class becomes Outer$2.class. This makes caching by source rather problematic and may not work at all with many JVM derived languages. To give an example:

object App {

    def isEven(i: Int) = i % 2 == 0

    def isOdd(i: Int) = i % 2 == 1

    def main(args: Array[String]): Unit = {
        val n = (1 to 10).toList
        n.filter(isEven).flatMap(i => n.filter(isOdd).map(j => i * j))
    }

}

when compiled with scalac App.scala you get:

App$$anonfun$main$1.class
App$$anonfun$main$2$$anonfun$apply$1.class
App$$anonfun$main$2$$anonfun$apply$2.class
App$$anonfun$main$2.class
App$.class
App.class

And you can see the issue of the order of the class things in the code can easily change. This would make tools like scons much less useful and possibly even problematic.

2

Trang chủ Giới thiệu Sinh nhật bé trai Sinh nhật bé gái Tổ chức sự kiện Biểu diễn giải trí Dịch vụ khác Trang trí tiệc cưới Tổ chức khai trương Tư vấn dịch vụ Thư viện ảnh Tin tức - sự kiện Liên hệ Chú hề sinh nhật Trang trí YEAR END PARTY công ty Trang trí tất niên cuối năm Trang trí tất niên xu hướng mới nhất Trang trí sinh nhật bé trai Hải Đăng Trang trí sinh nhật bé Khánh Vân Trang trí sinh nhật Bích Ngân Trang trí sinh nhật bé Thanh Trang Thuê ông già Noel phát quà Biểu diễn xiếc khỉ Xiếc quay đĩa Dịch vụ tổ chức sự kiện 5 sao Thông tin về chúng tôi Dịch vụ sinh nhật bé trai Dịch vụ sinh nhật bé gái Sự kiện trọn gói Các tiết mục giải trí Dịch vụ bổ trợ Tiệc cưới sang trọng Dịch vụ khai trương Tư vấn tổ chức sự kiện Hình ảnh sự kiện Cập nhật tin tức Liên hệ ngay Thuê chú hề chuyên nghiệp Tiệc tất niên cho công ty Trang trí tiệc cuối năm Tiệc tất niên độc đáo Sinh nhật bé Hải Đăng Sinh nhật đáng yêu bé Khánh Vân Sinh nhật sang trọng Bích Ngân Tiệc sinh nhật bé Thanh Trang Dịch vụ ông già Noel Xiếc thú vui nhộn Biểu diễn xiếc quay đĩa Dịch vụ tổ chức tiệc uy tín Khám phá dịch vụ của chúng tôi Tiệc sinh nhật cho bé trai Trang trí tiệc cho bé gái Gói sự kiện chuyên nghiệp Chương trình giải trí hấp dẫn Dịch vụ hỗ trợ sự kiện Trang trí tiệc cưới đẹp Khởi đầu thành công với khai trương Chuyên gia tư vấn sự kiện Xem ảnh các sự kiện đẹp Tin mới về sự kiện Kết nối với đội ngũ chuyên gia Chú hề vui nhộn cho tiệc sinh nhật Ý tưởng tiệc cuối năm Tất niên độc đáo Trang trí tiệc hiện đại Tổ chức sinh nhật cho Hải Đăng Sinh nhật độc quyền Khánh Vân Phong cách tiệc Bích Ngân Trang trí tiệc bé Thanh Trang Thuê dịch vụ ông già Noel chuyên nghiệp Xem xiếc khỉ đặc sắc Xiếc quay đĩa thú vị
Trang chủ Giới thiệu Sinh nhật bé trai Sinh nhật bé gái Tổ chức sự kiện Biểu diễn giải trí Dịch vụ khác Trang trí tiệc cưới Tổ chức khai trương Tư vấn dịch vụ Thư viện ảnh Tin tức - sự kiện Liên hệ Chú hề sinh nhật Trang trí YEAR END PARTY công ty Trang trí tất niên cuối năm Trang trí tất niên xu hướng mới nhất Trang trí sinh nhật bé trai Hải Đăng Trang trí sinh nhật bé Khánh Vân Trang trí sinh nhật Bích Ngân Trang trí sinh nhật bé Thanh Trang Thuê ông già Noel phát quà Biểu diễn xiếc khỉ Xiếc quay đĩa
Thiết kế website Thiết kế website Thiết kế website Cách kháng tài khoản quảng cáo Mua bán Fanpage Facebook Dịch vụ SEO Tổ chức sinh nhật