Recently, I ran into a small, but interesting, issue in a new
open-source project that I am working
on: RxnWeaver.
It is being developed in Go. Inspired by a few other projects
that I follow, I set up Travis CI for this project.
I do the development
in my fork of the
project, before raising periodic (or need-based) pull requests
to the main repository.
In a particular commit, I happened to add a few exported constants
to a package (let us call this Package A) in the
repository. In the next commit, I added code that depended on
some of those constants, to a different package (let us call
this Package B) in the repository. As usual, I pushed the
commits to my GitHub fork after making sure that the code was
formatted with go fmt and that the tree builds
without errors. I raised a pull request to the main repository,
and the fun began!
Travis CI reported failure saying that the build did not complete
successfully. A little investigation revealed the cause. In
Package B, references to Package A use the
official github.com/RxnWeaver/RxnWeaver import paths.
The official version of the package there, however, has the old
set of constants that does not include the new and required ones.
Therefore, Package B could not be built.
The commit, though, that was intended to update Package A was part
of the same pull request!
Of course, it was easy to fix, but the simple lesson is: if you do
not want to have to use the command line and some git
trickery, do not forget to raise a pull request (and have it
merged) for each piece that could be used as a dependency in a
different Go package!
As part of a consulting engagement, I accepted a project to develop some statistical inference models in the area of drug (medicine) repositioning. Input data comprises three sets of associations: (i) between drugs and adverse effects, (ii) between drugs and diseases, and (iii) between drugs and targets (proteins). Using drugs as hub elements, associations are inferred between the other three kinds of elements, pair-wise.
The actual statistics computed vary from simple measures such as sensitivity (e.g. how sensitive is a given drug to a set of query targets?) and clustering coefficients of the multi-mode graph, to construction of rather complex confusion matrices, computation of measures such as Matthews Correlation Coefficient, to construction of generalised profile vectors for drugs, diseases, etc. Accordingly, the computational intensity varies considerably across parts of the models.
For the size of the test subset of input data, the in-memory graph of direct and transitive associations currently has about 15,000 vertices and over 14,000,000 edges. This is expected to grow by two orders of magnitude (or more) when the full data set is used for input.
Programming Language
I had some temptation initially to prototype the first model (or two) in a language like Ruby. Giving the volume of data its due weight though, I decided to use Ruby for ad hoc validation of parts of the computations, with coding proper happening in a faster, compiled language. I have been using Java for most of my work (both open source as well as for clients). However, considering the fact that statistics instances are write-only, I hoped that Go could help me make the computations parallel easily[1].
My choice of Go caused some discomfort on the part of my client's programmers, since they have to maintain the code down the road. No serious objections were raised nevertheless. So, I went ahead and developed the first three models in Go.
Practical Issues With Go
The Internet is abuzz with success stories involving Go; there isn't an additional perspective that I can add! The following are factors, in no particular order, that inhibited my productivity as I worked on the project.
No Set in the Language
Through (almost) every hour of this project, I found myself needing an efficient implementation of a set data structure. Go does not have a built-in set; it has arrays, slices and maps (hash tables). And, Go lacks generics. Consequently, whichever generic data structure is not provided by the compiler can not be implemented in a library. I ended up using maps as sets. Everyone who does that realises the pain involved, sooner than later. Maps provide uniqueness of keys, but I needed sets for their set-like properties: being able to do minus, union, intersection, etc. I had to code those in-line every time. I have seen several people argue vehemently (even arrogantly) in golang-nuts that it costs just a few lines each time, and that it makes the code clearer. Nothing could be further from truth. In-lining those operations has only reduced readability and obscured my intent. I had to consciously train my eyes to recognise those blocks to mean union, intersection, etc. They also were very inconvenient when trying different sequences of computations for better efficiency, since a quick glance never sufficed!
Also, I found the performance of Go maps wanting. Profiling showed that get operations were consuming a good percentage of the total running time. Of course, several of those get operations are actually to check for the presence of a key.
No BitSet in the Standard Library
Since the performance of maps was dragging the computations back, I investigated the possibility of changing the algorithms to work with bit sets. However, there is no BitSet or BitArray in Go's standard library. I found two packages in the community: one on code.google.com and the other on github.com. I selected the former both because it performed better and provided a convenient iteration through only the bits set to true. Mind you, the data is mostly sparse, and hence both these were desirable characteristics.
Incidentally, both the bit set packages have varying performance. I could not determine the sources of those variations, since I could not easily construct test data to reproduce them on a small scale. A well-tested, high performance bit set in the standard library would have helped greatly.
Generics, or Their Absence
The general attitude in Go community towards generics seems to have degenerated into one consisting of a mix of disgust and condescension, unfortunately. Well-made cases that illustrate problems best served by generics, are being dismissed with such impudence and temerity as to cause repulsion. That Russ Cox' original formulation of the now-famous tri-lemma is incomplete at best has not sunk in despite four years of discussions. Enough said!
In my particular case, I have six sets of computations that differ in:
types of input data elements held in the containers, and upon which the computations are performed (a unique combination of three types for each pair, to be precise),
user-specified values for various algorithmic parameters for a given combination of element types,
minor computational steps and
types (and instances) of containers into which the results aggregate.
These differences meant that I could not write common template code that could be used to generate six versions using extra-language tools (as inconvenient as that already is). The amount of boiler-plate needed externally to handle the differences very quickly became both too much and too confusing. Eventually, I resorted to six fully-specialised versions each of data holders, algorithms and results containers, just for manageability of the code.
This had an undesirable side effect, though: now, each change to any of the core containers or computations had to be manually propagated to all the corresponding remaining versions. It soon led to a disinclination on my part to quickly iterate through alternative model formulations, since the overhead of trying new formulations was non-trivial.
Poor Performance
This was simply unexpected! With fully-specialised versions of graph nodes, edges, computations and results containers, I was expecting very good performance. Initially, it was not very good. In single-threaded mode, a complete run of three models on the test set of data took about 9 minutes 25 seconds. I re-examined various computations. I eliminated redundant checks in some paths, combined two passes into one at the expense of more memory, pre-identified query sets so that the full sets need not be iterated over, etc. At the end of all that, in single-threaded mode, a complete run of three models on the test set of data took about 2 minutes 40 seconds. For a while, I thought that I had squeezed it to the maximum extent. And so thought my client, too! More on that later.
Enhancement Requests
At that point, my client requested for three enhancements, two of which affected all the six + six versions of the models. I ploughed through the first change and propagated it through the other eleven specialised versions. I had a full taste of what was to come, though, when I was hit with the realisation that I was yet working on Phase 1 of the project, which had seven proposed phases in all!
Back to Java!
I took a break of one full day, and did a hard review of the code (and my situation, of course). I quickly identified three major areas where generics and (inheritance-based) polymorphism would have presented a much more pleasant solution. I had already spent 11 weeks on the project, the bulk of that going into developing and evaluating the statistical models. With the models now ready, I estimated that a re-write in Java would cost me about 10 working days. I decided to take the plunge.
The full re-write in Java took 8 working days. The ease with which I could model the generic data containers and results containers was quite expected. Java's BitSet class was of tremendous help. I had some trepidation about the algorithmic parts. However, they turned out to be easier than I anticipated! I made the computations themselves parts of formally-typed abstract classes, with the concrete parts such as substitution of actual types, the user-specified parameters and minor variations implemented by the subclasses. Conceptually, it was clear and clean: the base computations were easy to follow in the abstract classes. The overrides were clearly marked so, and were quite pointed.
Naturally, I expected a reduction in the size of the code base; I was not sure by how much, though. The actual reduction was by about 40%. This was nice, since it came with the benefit of more manageable code.
The most unexpected outcome concerned performance: a complete run of the three models on the test set of data now took about 30 seconds! My first suspicion was that something went so wrong as to cause a premature (but legal) exit somewhere. However, the output matched what was produced by the Go version (thanks Ruby), so that could not have been true. I re-ran the program several times, since it sounded too good to be true. Each time, the run completed in about 30 seconds.
I was left scratching my head. My puzzlement continued for a while, before I noticed something: the CPU utilisation reported by /usr/bin/time was around 370-380%! I was now totally stumped. conky showed that all processor cores were indeed being used. How could that be? The program was very much single-threaded.
After some thought and Googling, I saw a few factors that potentially enabled a utilisation of multiple cores.
All the input data classes were final.
All the results classes were final, with all of their members being final too.
All algorithm subclasses were final.
All data containers (masters), the multi-mode graph itself, and all results containers had only insert and look-up operations performed on them. None had a delete operation.
Effectively, almost all of the code involved only final classes. And, all operations were append-only. The compiler may have noticed those; the run-time must have noticed those. I still do not know what is going on inside the JRE as the program runs, but I am truly amazed by its capabilities! Needless to say, I am quite happy with the outcome, too!
Update: As several responses (both here and on Hacker News) stated, Java's multi-threaded GC appears to be primary reason for the utilisation of all the processor cores.
Conclusions
If your problem domain involves patterns that benefit from type parameterisation or[2] polymorphism that is easily achievable through inheritance, Go is a poor choice.
If you find your Go code evolving into having few interfaces but many higher-order functions (or methods) that resort to frequent type assertions, Go is a poor choice.
Go runtime can learn a trick or two from JRE 7 as regards performance.
These may seem obvious to more informed people; but to me, it was some enlightenment!
[1] I tried Haskell and Elixir as candidates, but nested data holders with multiply circular references appear to be problematic to deal with in functional languages. Immutable data presents interesting challenges when it comes to cyclic graphs! The solutions suggested by the respective communities involved considerable boiler-plate. More importantly, the resulting code lost direct correspondence with the problem's structural elements. Eventually, I abandoned that approach.↩
Firstly, I am surprised at the amount of traffic my previous post has generated. I had over a thousand visitors within a few hours: redirected from Google, Reddit, Hacker News, etc.! I wonder how may of them would have read the post had it been the opposite way, i.e., a declaration that I was adopting Go for the next version of my chemistry product? It appears to demonstrate an intriguing aspect of human nature!
Plug-in
Several people have suggested various ways in which out-of-process plug-ins are better. Some of these suggestions arose probably because of me not describing a plug-in adequately. I tried to remedy the situation in individual responses. I am collecting some of those points hereunder.
I am looking at plug-ins for at least the following benefits. In all cases, the said plug-in could potentially be supplied by me, a third-party developer or the user herself.
Substitute an algorithm for another.
Substitute an algorithm implementation for another.
Add a new algorithm not originally shipped with the product.
Add a calculator or a transformer as a hook in a particular processing step.
Performance
My application has to process millions (sometimes tens of millions) of molecules per run. A large number of them are processed by the proposed plug-ins. The number reduces with each advancing stage of processing, owing to elimination in each stage. The load is, hence, lower towards the tail of the work flow. But, upstream plug-ins are invoked for most molecules.
An out-of-process plug-in will require the following steps for communication:
serialisation of input in the main program,
deserialisation of input in the plug-in,
serialisation of output in the plug-in, and
deserialisation of output in the main program.
The above steps are in addition to the unavoidable protocol handshake for each request. Evidently, the larger the amount of data that needs to be exchanged between the main program and the plug-in, the slower the above process will be. Let us look at the data structure that will get exchanged the highest in my application, viz.,Molecule. It has the following information, at a minimum:
a unique identifier,
a list of atoms, where each atom has:
a unique identifier,
element type,
spatial coordinates,
net charge,
stereo configuration,
number of implicit H atoms attached to it,
aromaticity,
list of rings it is a part of,
whether it is a bridgehead,
…
a list of bonds, where each bond has:
a unique identifier,
the atoms it joins,
order: single, double, triple, aromatic, …,
stereo configuration,
aromaticity,
list of rings it is a part of,
…
a list of rings with their own properties,
a list of components,
a list of functional groups,
… .
It may be possible to have a protocol to allow a plug-in to declare the subset of the above data that it actually needs. However, checking the protocol and selectively serialising the data has a cost itself.
On the contrary, an in-memory plug-in accesses the object using a pointer, with just a transfer of ownership but no transfer of data.
Manageability
External plug-ins also raise the subject of their life cycle management and resolution. The questions that need to be addressed include the following.
When is a plug-in activated? Together with the main program? On demand?
Should plug-ins die with the main program? How do we handle the main program terminating abnormally?
How long should a plug-in continue to run, if idle?
How should zombie plug-ins be handled?
If socket-based, how should port numbers be managed?
How should multiple plug-ins providing the same capability be resolved? How about versions?
While in-memory plug-ins do not automatically solve all the above, they do eliminate some of them easily.
Interesting Options Suggested
An interesting option that was suggested was to package the Go binary distribution as part of my product. Then, when a plug-in is downloaded, the main program itself could be re-compiled and re-linked to include the plug-in. This is a possibility. Some infrastructure code has to be written, though.
Another family relates to embedding a scripting language. This is another possibility. However, it is neither fair nor acceptable to force third-party plug-ins to have to always suffer (the relatively) lower performance because of the scripting language itself. This may become very important if the plug-in is intended for use in the initial stages of the work flow.
The Need for a Java API
Independent of the above, there is no straight-forward way to provide a Java API on the top of an application written in Go. This issue remains unaddressed.
Considering the number of hits this post continues to have on a weekly basis, I should point out that a follow-up post can be found at A Difficult Decision - 2.
I like Go as a language. It is a rare, good balance between simplicity, power and expressiveness. I was so impressed with it as to begin the development of the next version of my chemistry product in it. There were minor hiccups, but I made progress, and wrote as many as about 12,000 lines of code (includes comments).
Then, I paused.
Now, about two months later, I have decided to shift the development of the new version of my product to … Java. If that sounds anti-climactic, so it is!
There are two significant reasons for this decision.
Plug-ins
The intended users of my product (and its API) are organic chemists and cheminformatics programmers. In practice, several organic chemists neither program themselves nor have programming assistants.
My product has several basic capabilities built into it. These are exposed as callable or configurable algorithms. What I also want to do is enable the user to extend the capabilities of the product through plug-ins.
In the Java land, this is well-studied problem, with more than one established way of solving. In the Go scenario, on the other hand, this is not straight-forward. Go programs are linked statically. There are requisites to enable plug-ins.
What would I need to do?
Supply the statically-linked full product.
Supply *.a files of my product's packages, and *.o files of the driver part.
What would my customer need to do?
Install gcc and family and their dependencies. [Once; update: not needed after Go1. Thanks Andrew Gerrand!]
Install (and maintain) the Go development environment. [Once, at least after Go1.]
Download the plug-in package file .a, and place it in a designated directory.
It could, with programmers. It, unfortunately, will not, with chemists. For many chemists, the exertion of supplying input to a computer program and reading its output using a suitable viewer, is a considerable condescension. Should I ask them to compile code, and that too for each plug-in, typical chemists will laugh me out of court!
The API Consumption Issue
Most foot soldier programmers at pharmaceutical companies, biotechnology companies and informatics services providers have graduated in the last ten years, or so. Most of them learned Java as their first language, and have used it the most. Naturally, they tend to gravitate towards solutions with a published Java API.
Chemistry product companies acknowledge this. Almost all of them provide a Java API for their products and libraries.[2] Those that have remained with a non-Java API are certainly not the leaders today!
Therefore, for me to appeal to those programmers, I must provide a fully capable Java API. My prospects of commercial success depend on that factor in no minor way.
Conclusion
Accordingly, I have shifted the product to Java. My team and I have begun re-designing and re-implementing the legacy product — this time to suit a more Java-like style; to appeal to Java audience!
[1] goinstall does not help, since it needs source code to be available. Plug-in authors may not be willing to publish source code.↩
[2] If they cannot provide a proper Java API, they at least provide a JNI wrapper.↩
N.B. This is draft version 2 of a work in progress. It is intended to help me understand the subject better, elicit comments from more knowledgeable people, and elucidate the nuances of generics in Go's context. The following proposal makes use of a syntax for ease of presentation. The syntax is, however, not intended to be the primary subject, nor is it intended to represent the complexities or possible complications of the underlying mechanisms.
Generics
In Go, there are fundamental differences between built-in types and user-defined types. When we try to come up with mechanisms that help treat both of them on a completely equal footing, they seem to stretch the language along difficult dimensions. Particularly troublesome are issues such as the following.
Built-in types (like int and float64) do not have methods; they only have a pre-defined set of operators. Users cannot redefine those operators, nor can they define new operators.
Operator overloading is not allowed, preventing user-defined types from exhibiting operator compatibility with the built-in types.
Operators are not methods themselves, and hence do not allow a symmetry between the built-in operations and the user-defined.
Rather than explore possible solutions to the above, the current proposal accepts them as facts that will not change. In other words, the inherent differences between built-in types and user-defined types are preserved, and no unification is attempted. By adding one new concept (that of , see hereunder) to the language, backwards-compatibility can be maintained even as generics are introduced into the language.
This proposal considers the following points as the base requirements for a working model of generics in Go.
Enabling generics for built-in types as well as user-defined types.
Avoiding boxing/unboxing and implicit casts.
Preserving Go's (package-level) separate compilation model.
Maintaining reasonably high compilation speed.
Promoting (generated) code sharing.
Allowing for reflection of generics.
Storage Classes
Types T1 and T2 belong to the same if elements of T1 can either be assigned to those of T2, or converted to T2, and vice versa. Please note that only conversion is considered, not a type assertion.
Type Classes
Types T1 and T2 belong to the same if the following are true:
T1 and T2 belong to the same storage class and
every operation defined on elements of T1 is also defined on those of T2, and vice versa.
As per the above definition, for instance, int32 and int64 belong to the same type class, while int64 and complex64 do not. Thus, type classes play the equivalent role of interfaces for built-in types. However, type classes do not introduce new concrete types.
Predefined Type Classes
Go language defines a set of predefined type classes for numeric types, as enumerated here.
Integer This type class encompasses the following built-in types: int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64 and byte.
Float This type class encompasses the following built-in types: float32 and float64.
Real This type class composes the type classes Integer and Float above.
Complex This type class encompasses the following built-in types: complex64 and complex128.
In addition, the set of all named interface types in a program constitutes a single type class. This class is unnamed, and is not accessible to the users; it is defined solely for the purposes of implementation.
It is not possible to define new type classes.
Inability to define new type classes may be a limitation. However, my current thinking is that the capability to define such may not be a pressing requirement. In any case, such a capability cannot be provided until the mechanisms governing the composition of type classes is unambiguously understood.
Generic Types
A type G with a type parameter T is said to be in T. It is denoted by G<T>. It is allowed for a generic type to be generic in more than one type. Where multiple type parameters need to be specified together, they are comma-separated. Go's generic types are reified, and are amenable to introspection at run time.
A type argument T can be any valid, assignable (within the package scope of applicability) Go type, with the exception of interface{}. The semantics of interface{} interfere with the compile-time checks that generics require. Moreover, the choice of interface{} for a type expressly avoids compile-time specification of a concrete type on the part of the user. In a sense, the compile-time mechanisms of generics constitute the conjugate space of that constituted by the run-time mechanisms of interface{}.
The semantics of generic types are dependent on the type parameter constraints (described hereunder) and actual type arguments; in particular, value types and reference types lead to different semantics.
A type parameter T can appear in the following positions (or sites):
type parameter declaration,
variable type specification,
function/method parameter type specification and
allocation.
Type Parameter Declaration
Type parameter declaration makes a new type parameter available for subsequent use within the applicable scope. Such a declaration does not introduce a new concrete type. It is illegal to declare a type parameter that is unused within that scope; similarly, it is illegal to use a type parameter that is not declared in that scope or any of its enclosing scopes. Type parameters are not shadowed. Hence, it is illegal to declare a type parameter that is already in scope, again.
A generic type G<T> with a specification for T introduces a new concrete type. This new concrete type behaves in full compliance with the rules governing normal Go types, including those for type inference, with explicit exceptions where noted.
Given a generic type G that is dependent on type parameters T1, ..., Tm, it is possible to construct a generic type H that is the same as G with n, n<m type parameters instantiated. This results in H being generic in m-n parameters.
type HashTable<K, V> struct {
...
}
type HashSet<K> HashTable<K, bool>
Variable Type Specification
Using a generic type as the type of a variable requires specification of concrete types for the generic type's type parameters. The number of specified type arguments should match that in the generic type's declaration. It is illegal to leave any type parameter uninstantiated, unless such parameter is in scope from an enclosing declaration.
var h1 HashTable<string, float64>
var h2 HashTable<string, T>
It is allowed to specify a type parameter as the type of a variable, as long as the type parameter has a valid declaration in the current scope.
It is illegal for a naked type parameter to itself act as a generic type. For instance, the following is illegal.
type A<T, U> struct {
x T<U>
}
Function/Method Parameter Type Specification
A function or a method can utilize type parameters available in scope. The function/method will then be generic in those type parameters utilized.
func (h *HashTable<K, V>) At(k K) V {
...
}
A function/method may itself be generic in one type parameter or more. This is independent of its being generic in one type parameter or more from an outer scope.
func modulus<T>(v T) float64 {
...
}
Allocation
In the presence of type parameters, allocation requirements may be determined at run time. The type class of the type parameter determines the allocation mechanism. Also, value types and reference types have different semantics, as described later.
type S struct {
...
}
var v = new(HashTable<string, S>)
An Example
The following example illustrates the above.
type Vector<T> []T
func (v *Vector<T>) Push(el T) {
*v = append(*v, el)
}
func (v *Vector<T>) Map<U>(mapper func (T) U) *Vector<U> {
u := new(Vector<U>)
for _, el := range *v {
u.Push(mapper(el))
}
return u
}
Constrained Generic Types
Since an unconstrained generic type cannot assume anything specific about the nature of the types eventually instantiated into its type parameters, the set of possible operations on the elements of such types is very small. Constraints can improve the situation by allowing a wider set of operations.
A type parameter can optionally be constrained to only those types that satisfy a particular interface. This enables all the methods from the interface to be utilized in the context of variables belonging to the applicable type parameter.
type Comparable<E> interface {
Compare(E) int
}
type OrderedSet<E : Comparable<E>> struct {
...
}
The above declaration allows ordered sets to be constructed over any type whose elements are ordered according to the Compare() relation.
Similarly, a type parameter can optionally be constrained to only those types that belong to a particular type class.
type Vector<T : Integer> []T
The above declaration allows a generic vector type to be written with algorithms that provide operations for any integral type.
Constraints on type parameters can appear at all sites except allocation sites. A constraint on a type parameter need only be specified when it is introduced for the first time. In the example above, for instance, the methods of OrderedSet need not repeat the constraint on E.
It is illegal to specify more than one constraint for a given type parameter.
Implementation Notes
Caveat: I do not understand the current Go compilers. Hence, the following is only a hint. More knowledgeable people should be able to comment on the naïveness, the feasibility and the complexity of implementing such a scheme as what is suggested here. I have represented some of the logic in the form of a high-level pseudocode, in the hope of its making it easier to discuss the same.
The following implementation notes cover various aspects of compilation, linking and reflection.
Compilation of Generics
In Go, compilation of a generic type generates two outputs: (a) generic type metadata and (b) partially compiled form of the generic.
Generic Type Metadata
At least the following metadata is generated for a given generic type, during compilation. This information is packed into a structure, which we refer to as , that is amenable to reflection.
unique identifier of the generic,
number of type parameters,
type handles of all type parameters,
function descriptors of all methods and
a pointer to the method table.
A similar type handle is generated for each type parameter. It contains at least the following metadata.
unique identifier of the type parameter,
a slot to hold the size of an instantiated type,
any constraint specified for it and
a pointer to its entry in the applicable function/method table.
In the case of functions and methods, a is generated. As with a type handle, this too is amenable to reflection. It contains at least the following metadata.
unique identifier of the function/method and
type handles of all type parameters.
It should be noted that the implementation distinguishes function-specific, or method-, type parameters from those declared in any applicable enclosing generic type.
If the generic type or function is exported, the above information is exported from the package of origin, and is available to the compiler, the linker and the runtime.
Partial Compilation
Each type parameter of the generic type serves as a slot that is filled for each instance of the generic type. A pointer to the metadata structure mentioned above fills this slot.
When compiling code, the size of the involved data types needs to be known. The following mechanism is used in order to arrive at one, for each type parameter.
When a generic type is compiled, the compiler first checks to see if its type parameters are constrained. For each type parameter constrained to a type class, the size of its elements is chosen to be that of the widest type (not in the variance sense, since Go does not support classical inheritance, but in the storage sense) in that class.
For each type parameter that is either constrained to an interface or unconstrained, the size is looked up from its type handle. This size is only a placeholder; it is filled, and may differ, for each instantiation of the generic type. This is applicable to all user-defined types supplied as value type arguments to any generic type.
compileGeneric() {
if generic is a struct or an interface {
generate type handle metadata placeholder
} else {
generate a function descriptor placeholder
}
for each type-parameter {
if type-parameter is constrained {
if constrained to a type-class {
use the size of the widest type in class
point to the predefined pseudo-method-set of this type-class
} else {
set a mark to obtain the size from type handle
setup, or point to, the method set of the interface
}
} else {
set a mark to obtain the size from type handle
point to a predefined empty method set
}
}
type check the code as per the above type handles and method sets
generate Go AST using the above type handles and function descriptors
serialize the generated AST into the package binary
set pointers for functions, methods and method tables
}
Notes
The choice of an AST representation of the generic's code is with the intention of preserving the separate compilation model of Go. It also relieves us of the need to invent a new exportable intermediate language; we just use a standard part of the language. The package go/ast may have to be enhanced to support this requirement.
Please note that the generic code is type checked only once, in its package of origin. For each instantiation, the supplied type arguments are checked only against any specified constraints. This speeds up compilation, since there is no need for repeated full type checks. Even though we are using an approach opposite to that of C++ (where type checks are completely use-site based), we are not going anywhere near H-M type system.
The above mechanism - as described - is not space-efficient for type parameters constrained to type classes. For instance, a vector of bytes will occupy more than a byte per byte. This is a trade-off between monomorphizing (code bloat) and user data space-efficiency. However, this could be tuned.
This look up of the actual size from the corresponding type handle introduces a small run time overhead. However, it greatly facilitates code sharing. It may be possible, and even easy, to avoid this overhead through the use of some clever trick. I do not know enough!
I also have not thought through any possible repercussions of differently-instantiated recursive functions and methods. For instance, foo<T1>(x T1) may have a call to foo<T2>(y), where y has type T2.
Compilation of Client Code
Once the generic type is compiled as above, client code can be written to instantiate and use it.
Metadata
When the generic type is instantiated in client code, and type parameters are specified, those type arguments are substituted in a copy of the generic's metadata. The result is a new concrete type. For most type checking purposes, the generic origins of this new concrete type can be ignored, as long as the applicable constraints are taken into account.
It should be noted that the unique identifier of the generic can be used to trace this instance back to its generic type definition. Similarly for type parameters and their constraints. This filled copy of the metadata is available to the compiler, the linker and the runtime.
Instance Code Generation
Generation of machine code for a specific instance of the generic type depends on factors such as whether a supplied type argument is a value type or not, and whether it is constrained or not. The following high-level logic could help illustrate the mechanism.
generateClientCode() {
create a local copy of the generic type's metadata structure
fill the metadata with actual type arguments
if new concrete type in package dictionary {
register this type to use the corresponding code
return
}
if new concrete type type-class-compatible with an existing type {
register this type to use the corresponding code
return
}
read the serialized AST of the generic from the package of origin
for each type-argument {
type check against any applicable constraint
if type-argument is value-type {
handleValueType(type-argument)
} else {
handleReferenceType(type-argument)
}
}
generate binary code from the AST, filling type information
register new concrete type in package dictionary with pointer to generated code
}
//
handleValueType(type-argument) {
if type-argument belongs to a built-in type-class {
use the pre-selected (widest) element size
} else {
use the actual size of the given type
}
}
handleReferenceType(type-argument) {
use the predefined size of standard interface
}
The package dictionary referred to is always exported, i.e., it is available to the compiler, the linker and the runtime. This is independent of whether a particular generic instance is itself exported.
I currently think that the above exported information need not contain package namespace qualifiers. But, I could be wrong in thinking so, and namespace qualifiers may be necessary for some purpose.
In order to facilitate the process of looking up the size of a given value type at run time, a hidden parameter is inserted into all applicable function and method signatures. The corresponding type handle of the supplied type argument is passed into the said hidden parameter at run time. When multiple type arguments are involved, a single consolidated meta handle of their type handles is passed into the hidden parameter. This simplifies the internal signatures of the functions/methods.
Notes
Since Go does not support traditional inheritance, there is no transitive closure of types that we need to take into account. Thus, the amount of code explosion is much less than what it would be had transitive closure been applicable. It should be noted that this applies independent of whether generic metadata is used to reuse generated code or not. But, the combination of the two results in a big win over what causes code bloat otherwise.
A topic that is not discussed relates to partial specialization. It can be dealt with once the mechanisms governing the simpler cases are understood and established.
Linking
During linking, the generic metadata exported from various packages is checked to see if there are redundant copies of:
the same metadata in different packages and
the same code in different packages (library files).
A global dictionary is created by merging the above information from all packages. This global dictionary is constructed using the exported package-level dictionaries themselves. The detected redundant copies are not linked into the final executable.
Reflection Capabilities
Reflection of generic types at run time is possible. Type handles and function descriptors are utilized for this purpose. At least the following capabilities are provided through the reflection API.
Query if a type is generic.
Query the number of type parameters.
Query the number of type-specific parameters.
Query the number of function-specific (or method-) parameters.
Retrieve specific or all type handles.
Retrieve the information within a type handle or a function descriptor.
The specific form of the API can be determined later to suit the philosophy of the current reflection API.
N.B. This is a work in progress. It is intended to help me understand the subject better, elicit comments from more knowledgeable people, and elucidate the nuances of generics in Go's context. The following proposal makes use of a syntax for ease of presentation. The syntax is, however, not intended to be the primary subject, nor is it intended to represent the complexities or possible complications of the underlying mechanisms.
Background
In Go, there are fundamental differences between built-in types and user-defined types. When we try to come up with mechanisms that help treat both of them on a completely equal footing, it appears to be stretching the language along difficult dimensions. Particularly troublesome are issues such as the following.
Built-in types (like int and float64) do not have methods; they only have a pre-defined set of operators. Users cannot redefine those operators, nor can they define new operators.
Operator overloading is not allowed, preventing user-defined types from exhibiting operator compatibility with the built-in types.
Operators are not methods themselves, and hence do not allow a symmetry between the built-in operations and the user-defined.
Rather than explore possible solutions to the above, the current proposal accepts them as facts that will not change. In other words, the inherent differences between built-in types and user-defined types are preserved, and no unification is attempted. By adding one new concept (that of , see hereunder) to the language, backwards-compatibility can be maintained even as generics are introduced into the language.
Storage Classes
Types T1 and T2 belong to the same if elements of T1 can either be assigned to those of T2, or converted to T2, and vice versa. Please note that only conversion is considered, not a type assertion.
Type Classes
Types T1 and T2 belong to the same if the following are true:
T1 and T2 belong to the same storage class and
every operation defined on elements of T1 is also defined on those of T2, and vice versa.
As per the above definition, for instance, int32 and int64 belong to the same type class, while int64 and complex64 do not. Thus, type classes play the equivalent role of interfaces for built-in types. However, type classes do not introduce new concrete types.
Predefined Type Classes
Go language defines a set of predefined type classes for numeric types, as enumerated here.
Integer This type class encompasses the following built-in types: int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64 and byte.
Float This type class encompasses the following built-in types: float32 and float64.
Complex This type class encompasses the following built-in types: complex64 and complex128.
In addition, the set of all named interface types in a program constitutes a single type class. This class is unnamed, and is not accessible to the users; it is defined solely for the purposes of implementation. It is not possible to define new type classes.
Generic Types
A type G with a type parameter T is said to be in T. It is denoted by G<T>. It is allowed for a generic type to be generic in more than one type. Where multiple type parameters need to be specified together, they are comma-separated. Go's generic types are reified, and are amenable to introspection at run time.
A type parameter T can represent any valid, assignable Go type, with the exception of interface{}. The semantics of interface{} interfere with the compile-time checks that generics require. Moreover, the choice of interface{} for a type expressly avoids compile-time specification of a concrete type on the part of the user.
The semantics of generic types are dependent on the type parameters; in particular, value types and pointer types lead to different semantics. Go's generic types are reified, and additional type information is passed through hidden parameters (i.e., the dictionary approach). In the most general case, for efficiency reasons, a distinct reified copy may be generated for each instantiated type class. However, this is an implementation issue.
The dictionary approach incurs some run time cost, but usually does not need the compiler to be available at run time. In my opinion, it is desirable, since that preserves the current Go model.
A type parameter T can appear in the following positions (or sites):
type parameter declaration,
variable type specification,
function/method parameter type specification and
allocation.
Type Parameter Declaration
Type parameter declaration makes a new type parameter available for subsequent use within the applicable scope. Such a declaration does not introduce a new concrete type. It is illegal to declare a type parameter that is unused within that scope; similarly, it is illegal to use a type parameter that is not declared in that scope or any of its enclosing scopes. Type parameters are not shadowed. Hence, it is illegal to declare a type parameter that is already in scope, again.
A generic type G<T> with a specification for T introduces a new concrete type. This new concrete type behaves in full compliance with the rules governing normal Go types, including those for type inference, with explicit exceptions where noted.
Given a generic type G that is dependent on type parameters T1, ..., Tm, it is possible to construct a generic type H that is the same as G with n, n<m type parameters instantiated. This results in H being generic in m-n parameters.
type HashTable<K, V> struct {
...
}
type HashSet<K> HashTable<K, bool>
Variable Type Specification
Using a generic type as the type of a variable requires specification of concrete types for the generic type's type parameters. The number of specified type arguments should match that in the generic type's declaration. It is illegal to leave any type parameter uninstantiated.
var h1 HashTable<string, float64>
var h2 HashTable<string, T>
It is allowed to specify a type parameter as the type of a variable, as long as the type parameter has a valid declaration in the current scope.
Function/Method Parameter Type Specification
A function or a method can utilize type parameters available in scope. The function/method will then be generic in those type parameters utilized.
func (h *HashTable<K, V>) At(k K) V {
...
}
A function/method may itself be generic in one type parameter or more. This is independent of its being generic in one type parameter or more from an outer scope.
func modulus<T>(v T) float64 {
...
}
Allocation
In the presence of type parameters, allocation requirements may be determined at run time. The type class of the type parameter determines the allocation mechanism. Also, value types and pointer types have different semantics, as with non-generic types.
type S struct {
...
}
var v = new(HashTable<string, S>)
An Example
The following example illustrates the above.
type Vector<T> []T
func (v *Vector<T>) Push(el T) {
*v = append(*v, el)
}
func (v *Vector<T>) Map<U>(mapper func (T) U) *Vector<U> {
u := new(Vector<U>)
for _, el := range *v {
u.Push(mapper(el))
}
return u
}
Constrained Generic Types
Since an unconstrained generic type cannot assume anything specific about the nature of the types eventually instantiated into its type parameters, the set of possible operations on the elements of such types is very small. Constraints can improve the situation by allowing a wider set of operations. A type parameter can optionally be constrained to only those types that satisfy a particular interface. This enables all the methods from the interface to be utilized in the context of variables belonging to the applicable type parameter.
type Comparable<E> interface {
Compare(E) int
}
type OrderedSet<E : Comparable<E>> struct {
...
}
The above declaration allows ordered sets to be constructed over any type whose elements are ordered according to the Compare() relation.
Similarly, a type parameter can optionally be constrained to only those types that belong to a particular type class.
type Vector<T : Integer> []T
The above declaration allows a generic vector type to be written with algorithms that provide operations for any integral type.
Constraints on type parameters can appear at all sites except allocation sites. It is illegal to specify more than one constraint for a given type parameter.
One of my chemistry programs does a topological analysis of input molecules, printing a variety of information about them. It currently processes a standardized input file of about 20,000 molecules (19,944, to be precise) in about 18.5-19s. Considering the fact that the running time came down to that level from as much as 149s a few months ago, I am happy! A few key reasons for this improvement are:
I always have a temptation to use functional-style closures with collections. They suit my style of thinking. Accordingly, I wrote a custom collection – à lacontainer/vector – with several methods that take a closure. Examples include Collect (or Map), Detect (or Find), Select (or Filter) and TakeWhile. You may recollect that the no longer extant exp/iter used to provide some of those methods!
However, spurious closures are not yet getting converted into normal function calls by the Go compiler. So, the said replacements are required until the current work on escape analysis begins to yield positive results. It is rather inelegant, but has no alternative currently.
A related issue concerns inlining. As Ron Minnich remarked, sometimes manually inlining a function may produce a better result than what a set of sophisticated algorithms employed by the compiler may produce. Nevertheless, Russ Cox has confirmed that inlining is a high-priority item. So, we can expect to see some improvement in the coming months.
When both the above get addressed, I am sure that almost all Go programs stand to benefit significantly. It does depend on the kind of actual workload in the program, but a large class of programs should benefit automatically. I, for one, eagerly await that release!
The utility program stsearchsdf that I wrote about previously has a simple supervisor-workers setup for processing the input molecules in parallel. To begin with I have an iterator that reads the input SDF file, and returns the text of the next molecule, on each call to Next(). The input file is represented by a *bufio.Reader as shown here.
// SDFile represents an on-disk file in MDL's SDF format.
typeSDFilestruct {
f *os.File
}
// Init initializes an SDFile with the underlying file.
func (s *SDFile) Init(fn string, flag int, mode uint32) os.Error {
f, e := os.OpenFile(fn, flag, mode)
if e != nil {
return e
}
s.f = f
returnnil
}
// Close closes the underlying file.
func (s *SDFile) Close() {
s.f.Close()
}
// SdfIter is an iterator over SDFile.
typeSdfIterstruct {
r *bufio.Reader
}
Once we open the input SDF file, we can request for an iterator as follows.
// Iter creates and returns an iterator over the current SDF file's
// molecules.
func (s *SDFile) Iter() *SdfIter {
it := new(SdfIter)
it.r, _ = bufio.NewReaderSize(s.f, BUF_SIZE)
return it
}
This iterator utilizes the sequence of characters that separates consecutive molecules (the sequence is $$$$) to read the text corresponding to one molecule. It answers that text in the success scenario.
// Next reads the text of one molecule from the input buffered stream
// into a slice, and returns the same.
func (it *SdfIter) Next() (v string, b bool, err os.Error) {
sl := make([]string, 0, SLICE_SIZE)
for {
s, e := it.r.ReadSlice(REC_SEP_H)
if e != nil {
err = io.ErrUnexpectedEOF
return
}
sl = append(sl, string(s))
s, e = it.r.ReadSlice('\n')
if e != nil {
err = io.ErrUnexpectedEOF
return
}
sl = append(sl, string(s))
if REC_SEP_T == strings.TrimRightFunc(string(s), IsWspace) {
b = truebreak
}
}
v = strings.Join(sl, "")
return
}
This iterator is then made use of by the supervisor to read the molecules one at a time. Meanwhile, the supervisor creates the necessary communication channels. Since the nature of the actual task varies based on the user's request, the function that represents the task itself (simply denoted fn here) is injected into the supervisor loop function. opts.mx determines the number of goroutines to spawn, and is taken from the command line.
The supervisor keeps track of whether the entire input file is processed, whether the search result has been found, and the number of live worker goroutines. When there are no more worker goroutines, the supervisor loop itself can exit.
// Loop over the file.
found := false
eof := false
liveHelpers := opts.mx
L1: for {
if 0 == liveHelpers {
breakL1
}
When the result has been found or the input file is exhausted or there was an error, all workers are sent termination messages, which each of them acknowledges. The acknowledgement is shown later.
select {
case rmsg := <-rch:
if (-1 == rmsg.mn) && (!rmsg.b) {
liveHelpers--
continueL1
}
if !found && rmsg.b {
fmt.Println("-- Molecule number :", rmsg.mn, "\n")
fmt.Println(rmsg.v)
found = true
}
if found || eof {
tchs[rmsg.id] <- TMsg{-1, nil, nil}
continueL1
}
We are now ready for the task distribution itself. A defined number of molecules serves as the quantum task size. This grouping helps in reducing the number of messages sent back-and-forth over the channels.
var tmsg TMsg
tmsg.mn = i
tmsg.opts = opts
tmsg.v = make([]string, 0, TASK_SIZE)
k := 0
L2: for k = 0; k < TASK_SIZE; k++ {
v, b, _ := it.Next()
if !b {
breakL2// TODO: This may need better handling.
}
tmsg.v = append(tmsg.v, v)
i++
if i >= opts.to {
breakL2
}
}
if k > 0 {
tchs[rmsg.id] <- tmsg
} else {
tchs[rmsg.id] <- TMsg{-1, nil, nil}
continueL1
}
}
}
Once we exit the above loop, we can close the receiving channel since all the workers will have exited by then.
Now we take a look at one of the task functions that is run in a worker goroutine. As soon as a worker is created, it sends an idle message to the supervisor.
In response, the supervisor starts assigning work to it. The task handling loop is quite simple: it exits when it receives a termination request (which it first acknowledges); else it processes input data.
for {
tmsg := <-tch
if -1 == tmsg.mn {
rch <- RMsg{id, -1, "", false, nil}
close(tch)
break
}
var rmsg RMsg
rmsg.id = id
rmsg.b = falsefor k := 0; k < len(tmsg.v); k++ {
m := mdl.NewMolecule()
rmsg.e = m.ParseMol(strings.Split(tmsg.v[k], "\n", -1), skip)
ifnil == rmsg.e {
if m.HasAtoms(tmsg.opts.acount, tmsg.opts.comp) {
rmsg.mn = tmsg.mn + k + 1
rmsg.v = tmsg.v[k]
rmsg.b = truebreak
}
}
}
rch <- rmsg
}
Finally, depending on the actual outcome of the search, we show an appropriate message to the user.
The above is conceptually simple, and easy to understand. For production, we have to introduce the requisite error handling code, of course. But, Go makes it really easy to think about the task distribution structure at a reasonably high level. What is more, it allows us to implement such a structure in a straight-forward manner, too!
In November 2010, I wrote a small utility program - in Go - to view and search for molecules in an MDL SDF format file. I wrote about it here. As the program turned out to be quite useful, I cleaned it up, and made it more streamlined and modular. Here is its current version of the function that prints help text.
funcprintHelp() {
str := `
NAME
stsearchsdf - view, extract from and search an SDF file
SYNOPSIS - GENERAL FORM
stsearchsdf command [params]
SYNOPSIS - SPECIFIC FORMS
stsearchsdf help
stsearchsdf show in=filename [from=m] [to=n]
stsearchsdf copy in=filename out=filename [from=m] [to=n]
stsearchsdf searcha in=filename (symbol=count)... [comp=(eq|gt|lt)]
[from=m] [to=n] [mx=c]
stsearchsdf searcht in=filename (tag=tagname tagval=tagvalue)... [from=m]
[to=n] [mx=c]
stsearchsdf counta in=filename (symbol=count)... [comp=(eq|gt|lt)]
[from=m] [to=n] [mx=c]
stsearchsdf countt in=filename (tag=tagname tagval=tagvalue)... [from=m]
[to=n] [mx=c]
DESCRIPTION
stsearchsdf is a program that can be used to view or extract parts of an
SDF file. It can also be used to search an SDF file for molecules that
contain atoms of given elements in specified number, or those with
specified tag values.
COMMANDS
help
Display this help text, and exit.
show
Display the sequence of molecules in the specified serial number
range.
copy
Copy the sequence of molecules in the specified serial number range
to the given output file.
searcha
Search for and display the first matching molecule containing specified
numbers of given atom types.
searcht
Search for and display the first matching molecule containing specified
tag values.
counta
Display the number of matching molecules containing specified numbers
of given atom types.
countt
Display the number of matching molecules containing specified tag
values.
OPTIONS
in
Input SDF file name.
out
Output SDF file name. Any existing output file will be overwritten.
Applicable to only the command 'copy'.
from
Number of the molecule where processing should start. Defaults to 1.
to
Number of the molecule where processing should stop. Defaults to the
last molecule in the file.
symbol
Atomic symbol. This should be given in proper case, e.g. C, Na.
comp
The numeric comparison to be used. Use 'eq' for equality check, 'gt'
for greater than, and 'lt' for less than. Defaults to 'eq'.
tag
Name of the tag whose value follows.
tagval
Value of the tag field whose name this follows.
mx
Maximum number of threads to use for processing. Defaults to 2.
Applicable to only search and count commands.
`
fmt.Println(str)
}
After a hiatus of a few months, here is an update. What follows is an e-mail that I sent to my collaborators regarding the future technological direction of our organic chemistry product.
* * * *
Apart from resuming my work on importing available chemicals databases, etc., I have spent some time over the last few weeks studying recent developments in making programs multi-threaded, etc. I have written several multi-threaded programs in the past. Usually, the following are the important problems that arise in the context of multi-threaded programs:
identifying the variables (mutable state of data) that need to be accessed from more than one thread,
determining the locking mechanisms needed to protect simultaneous access to such variables from multiple threads, and
using appropriate synchronization mechanisms for proper control flow and cause-effect relationships.
In contrast to the model described above, an alternative approach was proposed by Hewitt, Bishop and Steiger in 1973. It is called the Actor model. It comprises independent objects (all objects are actors) communicating via messages. No other form of interaction exists between independent objects. This communication can be synchronous or asynchronous, but is usually asynchronous by default. Also, actors can create new actors. Actors can also maintain state, designating what their behavior should be in response to future incoming messages.
Yet another alternative was proposed in 1978 by Hoare, called Communicating Sequential Processes (CSP). While being similar in spirit to the Actor model (all objects are communicating processes), CSP differs in a few aspects. For instance, in the Actor model, the sender of a message needs to know the address of the receiver in order to send a message. In CSP, communication happens through channels. A sender need not be aware of the identity of the receiver at the other end of the channel. Also, in CSP, message-passing is in itself a synchronization mechanism: a sender cannot complete the process of sending a message unless the receiver is ready to accept one. In the Actor model, on the other hand, the `system' itself holds the messages that receivers are not yet ready to accept.
Please refer to http://en.wikipedia.org/wiki/Actor_model and http://en.wikipedia.org/wiki/Communicating_sequential_processes for introductory details.
It is pertinent to note that neither the Actor model nor CSP mandates conventional locks or globally-shared data! It is known in the industry that locking and lock-based synchronization are difficult to implement correctly, and even more difficult to scale up to multi-processor scenarios.
About 2-3 years ago, when I used Scala (http://www.scala-lang.org) for a few programs, I utilized the Actor model (provided in Scala through a standard library) to develop a framework for lock-less multi-threading that could utilize all the available processors. That was, of course, before I had heard of Akka. I employed it successfully for processing large amounts of data from the chemical databases of one of my clients.
Some time last year, I became aware of, and interested in, the Go (http://golang.org) language. These days, I use it for most new programs that I develop. Go implements CSP at the language level itself. I have found it to be very useful and convenient for developing multi-threaded programs that operate in a lock-less manner. I have recently written a utility program, using Go, to view and search for molecules in MDL's SDF file format.
Due to C and C++ not having an inherent notion of threading or message passing, there exists no straight-forward way of making ST's main program multi-threaded. We have to employ the Boost library or the MPI(Message-Passing Interface) library or (worse) POSIX threads directly or something similar, in order to wrap certain parts of the current main program. I remember that you looked at MPI (OpenMPI, in particular) several months ago
In my (rather limited) experience, both Boost libraries and MPI present steep learning gradients. They are large, their APIs are large, they are inherently complex, and the very amount of code they require us to write is significantly high. I am, in fact, more concerned about the effort needed to debug and maintain such code in the long run. In contrast, I have found Scala's Actor model and Go's CSP model to be welcome reliefs by way of their simplicity and ease of use.
Scala is a language implemented for the Java Virtual Machine (JVM). This, unfortunately, means that should we choose to employ Scala, our clients have to install Java in their Linux computers. Installation of Java is a decision outside our control, and is made by their respective IT departments.
Go, on the other hand, is similar to C and C++: it compiles to native code, and does not require a virtual machine. Consequently, we can give a new main program to our clients without asking them to install a JVM first. More importantly, we can easily call C code from within Go, making it feasible to re-use select parts of the current main program in a more flexible manner.
In view of the above, after several weeks of study and trials, I think that we should employ Go for making ST's main program multi-threaded. This applies to all areas of the main program that lend themselves to multi-threading, in general, and to the convergent synthesis problem, in particular.
I have written a small part of the new SDF import program in Go. Apart from a particular annoyance that I encountered (because of the absence of generics in Go), I should say that I was delighted to see how easily the program could be written multi-threaded to utilize both the processors in my computer. The effective increase in speed was about 50%, and I think that it can increase even further with some optimization.
The above may be unclear or confusing in parts. Please ask me questions if it is so. However, kindly note that this is an important decision that has to be made, and which cannot be reversed (that easily) later. Accordingly, the more the number of questions that we consider now, the better.
As part of my product, I have an available chemicals database (ACD) that is compiled from third-party catalogs. Such catalogs are typically in SDF file format, as specified by MDL. I ran into some issues when importing them, resulting in aborted runs. It was getting tedious to locate the troublesome molecule in the vastness of the SD files. After a while, the programmer in me woke up! The consequence is a small (~750 LoC) utility written in Go, which I blandly called sdf.
~ % sdf help
Usage:
sdf help
Prints this usage notes message, and exits.
sdf show in=file [from=m] [to=n]
Fetches and displays molecules from the file 'file',
starting with the molecule numbered 'm', optionally to the
molecule numbered 'n'. If 'from' is not specified, the
first molecule is used as the starting molecule. If 'to'
is not specified, all molecules until the end of file are
displayed. Specifically, to display only the m'th molecule,
you should specify 'from=m to=m'.
sdf copy in=file1 out=file2 [from=m] [to=n]
Similar to 'show' above, difference being that the output
is written to 'file2'. Any existing 'file2' will be
truncated.
sdf searcha in=file [from=m] [to=n] symbol=count [symbol=count] [mx=c]
Performs a search for the first molecule in the given range
that has the specified number of atoms of each element type.
The number of processor cores to use can be specified using
'mx'; default is 2.
sdf searcht in=file [from=m] [to=n] tag=name tagval=value [tag=name tagval=value] [mx=c]
Performs a search for the first molecule in the given range
that has the specified tags and values. The number of
processor cores to use can be specified using 'mx'; default
is 2.
I was thinking of filing an issue for the stat problem of Go for Windows. I was about to do that this evening, when I received my daily digest of Go mailing list. I found an announcement therein that a new release of Go was made yesterday night. The release notes said, to my disbelief, that the stat problem in Windows was solved!
I eagerly downloaded the new version for Windows, and re-compiled the SHA-1 summing program with it. I ran it, I admit, with some trepidation. It ran as it should (have) -- smoothly!
Now, that is some coincidence! Happy ending, thusly!
I wrote a small program to compute SHA-1 sums of directory trees, to easily identify and locate files occurring more than once. I wrote it in Linux using Go language. I tested it on large trees of sizes in the range 1-4 GB. I cross-checked the results - through random sampling - using `sha1sum', and I was happy! I then booted into Windows 7 VMware image and compiled it. Then began the trouble.
Go is not officially supported on Windows yet. There is an unofficial port that closely follows the official releases. It is clearly marked `experimental'. I should have realized that when an OS is not yet officially supported, the weakest areas would be OS interfaces. The inane dud that I am, I did not think enough!
While walking the directory tree, I was doing a `stat' of each entry: (a) to find whether it is a directory or a file, and (b) if it is a file, to know its size in bytes. In Windows, the `stat' call was failing. After some gymnastics, the program was walking the directory tree. But, some directories were getting recognized as files. In addition, I could not make it `stat' the entries whose full path contained spaces. I have exhausted all conventional mechanisms to no avail.
So, I give up! At least, for now. I have begun using Go for serious work. I have not encountered any issues with the runtime so far. But ... that is in Linux. From a few small programs that I run in both Linux and Windows, I see that there are no issues with Go ... as long as OS interfaces are not used in Windows!
My choice of language for this particular job was wrong, leading to this fruitless effort. Sigh!
