SemVer policies for ImageJ2 and Fiji

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

SemVer policies for ImageJ2 and Fiji

Stephan Saalfeld-2
Dear list,

a recent SPIM_Registration bug report on GitHub

https://github.com/bigdataviewer/SPIM_Registration/issues/10#issuecomment-79721014

resulted in a discussion about the correct way of assigning version
numbers to individual artifacts.

We have earlier settled to follow SemVer

http://semver.org/

which has reasonably clear guidelines under what circumstances to
increase which of three version counters.

<quote src="http://semver.org/">
1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backwards-compatible
manner, and
3. PATCH version when you make backwards-compatible bug fixes.
</quote>

It is unspecified, however, how version changes in dependency artifacts
affect the version of the consuming artifact, i.e. do they bubble or do
they not?  In my current understanding, there are two competing
objectives, deployment (advocated and executed by @dscho and @ctrueden),
and development (advocated and executed by @axtimwalde and @ctrueden, we
see that @ctrueden is ambivalent, @axtimwalde too, as always, just that
you know):

1.
Developer perspective: Don't bubble!  Developers that use the public API
of an artifact X in their code use SemVer to reason whether on upgrade
of X they will have to change their code or simply recompile with no
modification.  Dependency version bubbling would break this contract,
because it signalizes API incompatibility in X when a dependency Y of X
introduces incompatible changes, although this incompatibility does not
affect the public API of X.  Deployment of a set of artifacts following
this contract requires complete inspection of the entire set of
artifacts to guarantee consistency of the deployed version by other
means (maven pom tracing?) because the SemVer versions of individual
artifacts do not encode the necessary information.  I have the
impression that the developer perspective, ignoring it not being helpful
for deployment, was the driver of the SemVer specification and have a
preference for it because...

2.
Deployer perspective: Bubble?  Deployers could use SemVer to reason
whether a new version of an artifact has ANY incompatible changes
anywhere in its dependency tree.  This is useful to know when an
artifact in the tree can be released *without* considering its
dependents *and* dependencies (PATCH increase).  However, whenever the
version number signalizes incompatibility (MAJOR) or new features
(MINOR), further inspection of the entire dependency tree is required
because consistency cannot be derived from SemVer versions alone.  The
only definite conclusion that can be made from observing that an
artifact changes its MAJOR or MINOR version, i.e. becomes incompatible,
is that all dependents will need to be updated/ recompiled or that
there's a problem, consistency across artifacts cannot be guaranteed.
I.e. the bubbling scheme, at every individual artifact, sends a signal
when further inspection is required.  This information, however is
binary, and a single counter would suffice to do that.  If patch
counters are desired, one would need two.  The MAJOR and MINOR counters
are redundant.

Short:
Non-bubbling SemVer tells a developer whether her code will compile with
a given dependency artifact (PATCH and MINOR).  Bubbling SemVer tells a
deployer when a single artifact can be deployed without considering its
environment (PATCH).  Both things exclude each other.  In this setup, I
find the benefit for developers stronger and therefore prefer
non-bubbling SemVer.

An interesting animal are dependency management poms (BOMs) such as
pom-fiji or pom-mpicbg.  Their `public API' is the composite of  managed
dependencies and therefore it has to bubble the SemVer versions of the
managed dependencies.  This is different from artifacts that consume a
dependency, consuming and managing are different.  I may be wrong, but I
have the impression that these two things often get confused.

Please let me know your thoughts.

Thanks,
Stephan





_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel
Reply | Threaded
Open this post in threaded view
|

Re: SemVer policies for ImageJ2 and Fiji

Stephan Saalfeld-2
Dear ImageJ developers,

my earlier statement about the conclusion that can be drawn from
bubbling versions in a deployment context are wrong.

Counter proof: Artifact X fixes a bug in one of its public methods, and
accordingly increases the PATCH counter.  Artifact Y depends on artifact
X and knew about the bug and had a workaround in place to compensate for
it.  On upgrade of dependency version for X, it removes the workaround,
public API remains unchanged.  This is a patch and X had increased the
PATCH counter, so Y increases the PATCH counter.  Neither X nor Y can be
deployed independently.  The deployment system (or person) has to
inspect the entire dependency tree to calculate a correct state or a
conflict.  The same is true in the non-bubbling situation.  Ergo,
bubbling versioning has no advantage over non-bubbling versioning in a
deployment context.

This leads me to the conclusion that non-bubbling versioning is better
because it carries local information for developers that non-bubbling
versioning does not, i.e. in what way the API of the versioned artifact
was changed.

I expect to be wrong still and that I missed something important.

Looking forward to your responses.

Thanks,
Stephan



On Sat, 2015-03-14 at 23:12 -0400, Stephan Saalfeld wrote:

> Dear list,
>
> a recent SPIM_Registration bug report on GitHub
>
> https://github.com/bigdataviewer/SPIM_Registration/issues/10#issuecomment-79721014
>
> resulted in a discussion about the correct way of assigning version
> numbers to individual artifacts.
>
> We have earlier settled to follow SemVer
>
> http://semver.org/
>
> which has reasonably clear guidelines under what circumstances to
> increase which of three version counters.
>
> <quote src="http://semver.org/">
> 1. MAJOR version when you make incompatible API changes,
> 2. MINOR version when you add functionality in a backwards-compatible
> manner, and
> 3. PATCH version when you make backwards-compatible bug fixes.
> </quote>
>
> It is unspecified, however, how version changes in dependency artifacts
> affect the version of the consuming artifact, i.e. do they bubble or do
> they not?  In my current understanding, there are two competing
> objectives, deployment (advocated and executed by @dscho and @ctrueden),
> and development (advocated and executed by @axtimwalde and @ctrueden, we
> see that @ctrueden is ambivalent, @axtimwalde too, as always, just that
> you know):
>
> 1.
> Developer perspective: Don't bubble!  Developers that use the public API
> of an artifact X in their code use SemVer to reason whether on upgrade
> of X they will have to change their code or simply recompile with no
> modification.  Dependency version bubbling would break this contract,
> because it signalizes API incompatibility in X when a dependency Y of X
> introduces incompatible changes, although this incompatibility does not
> affect the public API of X.  Deployment of a set of artifacts following
> this contract requires complete inspection of the entire set of
> artifacts to guarantee consistency of the deployed version by other
> means (maven pom tracing?) because the SemVer versions of individual
> artifacts do not encode the necessary information.  I have the
> impression that the developer perspective, ignoring it not being helpful
> for deployment, was the driver of the SemVer specification and have a
> preference for it because...
>
> 2.
> Deployer perspective: Bubble?  Deployers could use SemVer to reason
> whether a new version of an artifact has ANY incompatible changes
> anywhere in its dependency tree.  This is useful to know when an
> artifact in the tree can be released *without* considering its
> dependents *and* dependencies (PATCH increase).  However, whenever the
> version number signalizes incompatibility (MAJOR) or new features
> (MINOR), further inspection of the entire dependency tree is required
> because consistency cannot be derived from SemVer versions alone.  The
> only definite conclusion that can be made from observing that an
> artifact changes its MAJOR or MINOR version, i.e. becomes incompatible,
> is that all dependents will need to be updated/ recompiled or that
> there's a problem, consistency across artifacts cannot be guaranteed.
> I.e. the bubbling scheme, at every individual artifact, sends a signal
> when further inspection is required.  This information, however is
> binary, and a single counter would suffice to do that.  If patch
> counters are desired, one would need two.  The MAJOR and MINOR counters
> are redundant.
>
> Short:
> Non-bubbling SemVer tells a developer whether her code will compile with
> a given dependency artifact (PATCH and MINOR).  Bubbling SemVer tells a
> deployer when a single artifact can be deployed without considering its
> environment (PATCH).  Both things exclude each other.  In this setup, I
> find the benefit for developers stronger and therefore prefer
> non-bubbling SemVer.
>
> An interesting animal are dependency management poms (BOMs) such as
> pom-fiji or pom-mpicbg.  Their `public API' is the composite of  managed
> dependencies and therefore it has to bubble the SemVer versions of the
> managed dependencies.  This is different from artifacts that consume a
> dependency, consuming and managing are different.  I may be wrong, but I
> have the impression that these two things often get confused.
>
> Please let me know your thoughts.
>
> Thanks,
> Stephan
>
>
>


_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel
Reply | Threaded
Open this post in threaded view
|

Re: SemVer policies for ImageJ2 and Fiji

Mark Hiner
Hi Stephan,

Thanks for starting this thread!

For those interested, there was further discussion in IRC today, and Curtis and I chatted briefly face-to-face.

Basically, we agree that semver must be limited in scope to the public API of a given project. Checking for dependency convergence must be handled separately. Bubbling versions through dependencies is not significantly meaningful.

Documentation on this (and other versioning issues) is on the wiki: http://imagej.net/Architecture#Versioning

>Artifact Y depends on artifact X and knew about the bug and had a workaround in place to compensate for it

As a side note, semver does not guarantee preservation of unintentional behavior. As long as behavior is fixed in a way that's backwards-compatible with public API, only patch versions need to change.

However, the idea is that a BOM will account for such a change, to avoid dependency skew.

Best,
Mark


On Sun, Mar 15, 2015 at 9:38 PM, Stephan Saalfeld <[hidden email]> wrote:
Dear ImageJ developers,

my earlier statement about the conclusion that can be drawn from
bubbling versions in a deployment context are wrong.

Counter proof: Artifact X fixes a bug in one of its public methods, and
accordingly increases the PATCH counter.  Artifact Y depends on artifact
X and knew about the bug and had a workaround in place to compensate for
it.  On upgrade of dependency version for X, it removes the workaround,
public API remains unchanged.  This is a patch and X had increased the
PATCH counter, so Y increases the PATCH counter.  Neither X nor Y can be
deployed independently.  The deployment system (or person) has to
inspect the entire dependency tree to calculate a correct state or a
conflict.  The same is true in the non-bubbling situation.  Ergo,
bubbling versioning has no advantage over non-bubbling versioning in a
deployment context.

This leads me to the conclusion that non-bubbling versioning is better
because it carries local information for developers that non-bubbling
versioning does not, i.e. in what way the API of the versioned artifact
was changed.

I expect to be wrong still and that I missed something important.

Looking forward to your responses.

Thanks,
Stephan



On Sat, 2015-03-14 at 23:12 -0400, Stephan Saalfeld wrote:
> Dear list,
>
> a recent SPIM_Registration bug report on GitHub
>
> https://github.com/bigdataviewer/SPIM_Registration/issues/10#issuecomment-79721014
>
> resulted in a discussion about the correct way of assigning version
> numbers to individual artifacts.
>
> We have earlier settled to follow SemVer
>
> http://semver.org/
>
> which has reasonably clear guidelines under what circumstances to
> increase which of three version counters.
>
> <quote src="http://semver.org/">
> 1. MAJOR version when you make incompatible API changes,
> 2. MINOR version when you add functionality in a backwards-compatible
> manner, and
> 3. PATCH version when you make backwards-compatible bug fixes.
> </quote>
>
> It is unspecified, however, how version changes in dependency artifacts
> affect the version of the consuming artifact, i.e. do they bubble or do
> they not?  In my current understanding, there are two competing
> objectives, deployment (advocated and executed by @dscho and @ctrueden),
> and development (advocated and executed by @axtimwalde and @ctrueden, we
> see that @ctrueden is ambivalent, @axtimwalde too, as always, just that
> you know):
>
> 1.
> Developer perspective: Don't bubble!  Developers that use the public API
> of an artifact X in their code use SemVer to reason whether on upgrade
> of X they will have to change their code or simply recompile with no
> modification.  Dependency version bubbling would break this contract,
> because it signalizes API incompatibility in X when a dependency Y of X
> introduces incompatible changes, although this incompatibility does not
> affect the public API of X.  Deployment of a set of artifacts following
> this contract requires complete inspection of the entire set of
> artifacts to guarantee consistency of the deployed version by other
> means (maven pom tracing?) because the SemVer versions of individual
> artifacts do not encode the necessary information.  I have the
> impression that the developer perspective, ignoring it not being helpful
> for deployment, was the driver of the SemVer specification and have a
> preference for it because...
>
> 2.
> Deployer perspective: Bubble?  Deployers could use SemVer to reason
> whether a new version of an artifact has ANY incompatible changes
> anywhere in its dependency tree.  This is useful to know when an
> artifact in the tree can be released *without* considering its
> dependents *and* dependencies (PATCH increase).  However, whenever the
> version number signalizes incompatibility (MAJOR) or new features
> (MINOR), further inspection of the entire dependency tree is required
> because consistency cannot be derived from SemVer versions alone.  The
> only definite conclusion that can be made from observing that an
> artifact changes its MAJOR or MINOR version, i.e. becomes incompatible,
> is that all dependents will need to be updated/ recompiled or that
> there's a problem, consistency across artifacts cannot be guaranteed.
> I.e. the bubbling scheme, at every individual artifact, sends a signal
> when further inspection is required.  This information, however is
> binary, and a single counter would suffice to do that.  If patch
> counters are desired, one would need two.  The MAJOR and MINOR counters
> are redundant.
>
> Short:
> Non-bubbling SemVer tells a developer whether her code will compile with
> a given dependency artifact (PATCH and MINOR).  Bubbling SemVer tells a
> deployer when a single artifact can be deployed without considering its
> environment (PATCH).  Both things exclude each other.  In this setup, I
> find the benefit for developers stronger and therefore prefer
> non-bubbling SemVer.
>
> An interesting animal are dependency management poms (BOMs) such as
> pom-fiji or pom-mpicbg.  Their `public API' is the composite of  managed
> dependencies and therefore it has to bubble the SemVer versions of the
> managed dependencies.  This is different from artifacts that consume a
> dependency, consuming and managing are different.  I may be wrong, but I
> have the impression that these two things often get confused.
>
> Please let me know your thoughts.
>
> Thanks,
> Stephan
>
>
>



_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel
Reply | Threaded
Open this post in threaded view
|

Re: SemVer policies for ImageJ2 and Fiji

Stephan Saalfeld-2
Hi Mark and everybody,

I am concerned about the PATCH counter and the notion of `unintended'
behavior (what's that in the first place).  A strict definition of the
public API would classify a bug-fix that changes the output of a method
from wrong to correct as a break, i.e. MAJOR upgrade.  If we would
follow this scheme, MINOR and PATCH increments are safe to consume, i.e.
do not change behavior.  If we, however, say that patches of
`unintended' behavior (bugfixes) lead to a PATCH counter increase, MAJOR
and PATCH increases are technically equivalent, and thanks to the SemVer
policy to roll PATCH on MINOR (1.1.5 > 1.2.0), MINOR does not exclude
PATCH, i.e. MAJOR == MINOR == PATCH == unsafe.  I certainly understand
that treating bugfixes as API-breakage is even farther from `romantic
versioning' than our current understanding of SemVer, but it seems the
only way to get at least something out of those numbers.

Please let me know what you think.

Best,
Stephan



On Mon, 2015-03-16 at 13:18 -0500, Mark Hiner wrote:

> Hi Stephan,
>
> Thanks for starting this thread!
>
> For those interested, there was further discussion in IRC
> <http://code.imagej.net/chatlogs/imagejdev?times=prefix&start-date=2015-03-16&end-date=2015-03-16#20150316T155138>
> today, and Curtis and I chatted briefly face-to-face.
>
> Basically, we agree that semver must be limited in scope to the public API
> of a given project. Checking for dependency convergence must be handled
> separately. Bubbling versions through dependencies is not significantly
> meaningful.
>
> Documentation on this (and other versioning issues) is on the wiki:
> http://imagej.net/Architecture#Versioning
>
> >Artifact Y depends on artifact X and knew about the bug and had a
> workaround in place to compensate for it
>
> As a side note, semver does not guarantee preservation of unintentional
> behavior. As long as behavior is fixed in a way that's backwards-compatible
> with public API, only patch versions need to change.
>
> However, the idea is that a BOM will account for such a change, to avoid
> dependency skew.
>
> Best,
> Mark
>
>
> On Sun, Mar 15, 2015 at 9:38 PM, Stephan Saalfeld <
> [hidden email]> wrote:
>
> > Dear ImageJ developers,
> >
> > my earlier statement about the conclusion that can be drawn from
> > bubbling versions in a deployment context are wrong.
> >
> > Counter proof: Artifact X fixes a bug in one of its public methods, and
> > accordingly increases the PATCH counter.  Artifact Y depends on artifact
> > X and knew about the bug and had a workaround in place to compensate for
> > it.  On upgrade of dependency version for X, it removes the workaround,
> > public API remains unchanged.  This is a patch and X had increased the
> > PATCH counter, so Y increases the PATCH counter.  Neither X nor Y can be
> > deployed independently.  The deployment system (or person) has to
> > inspect the entire dependency tree to calculate a correct state or a
> > conflict.  The same is true in the non-bubbling situation.  Ergo,
> > bubbling versioning has no advantage over non-bubbling versioning in a
> > deployment context.
> >
> > This leads me to the conclusion that non-bubbling versioning is better
> > because it carries local information for developers that non-bubbling
> > versioning does not, i.e. in what way the API of the versioned artifact
> > was changed.
> >
> > I expect to be wrong still and that I missed something important.
> >
> > Looking forward to your responses.
> >
> > Thanks,
> > Stephan
> >
> >
> >
> > On Sat, 2015-03-14 at 23:12 -0400, Stephan Saalfeld wrote:
> > > Dear list,
> > >
> > > a recent SPIM_Registration bug report on GitHub
> > >
> > >
> > https://github.com/bigdataviewer/SPIM_Registration/issues/10#issuecomment-79721014
> > >
> > > resulted in a discussion about the correct way of assigning version
> > > numbers to individual artifacts.
> > >
> > > We have earlier settled to follow SemVer
> > >
> > > http://semver.org/
> > >
> > > which has reasonably clear guidelines under what circumstances to
> > > increase which of three version counters.
> > >
> > > <quote src="http://semver.org/">
> > > 1. MAJOR version when you make incompatible API changes,
> > > 2. MINOR version when you add functionality in a backwards-compatible
> > > manner, and
> > > 3. PATCH version when you make backwards-compatible bug fixes.
> > > </quote>
> > >
> > > It is unspecified, however, how version changes in dependency artifacts
> > > affect the version of the consuming artifact, i.e. do they bubble or do
> > > they not?  In my current understanding, there are two competing
> > > objectives, deployment (advocated and executed by @dscho and @ctrueden),
> > > and development (advocated and executed by @axtimwalde and @ctrueden, we
> > > see that @ctrueden is ambivalent, @axtimwalde too, as always, just that
> > > you know):
> > >
> > > 1.
> > > Developer perspective: Don't bubble!  Developers that use the public API
> > > of an artifact X in their code use SemVer to reason whether on upgrade
> > > of X they will have to change their code or simply recompile with no
> > > modification.  Dependency version bubbling would break this contract,
> > > because it signalizes API incompatibility in X when a dependency Y of X
> > > introduces incompatible changes, although this incompatibility does not
> > > affect the public API of X.  Deployment of a set of artifacts following
> > > this contract requires complete inspection of the entire set of
> > > artifacts to guarantee consistency of the deployed version by other
> > > means (maven pom tracing?) because the SemVer versions of individual
> > > artifacts do not encode the necessary information.  I have the
> > > impression that the developer perspective, ignoring it not being helpful
> > > for deployment, was the driver of the SemVer specification and have a
> > > preference for it because...
> > >
> > > 2.
> > > Deployer perspective: Bubble?  Deployers could use SemVer to reason
> > > whether a new version of an artifact has ANY incompatible changes
> > > anywhere in its dependency tree.  This is useful to know when an
> > > artifact in the tree can be released *without* considering its
> > > dependents *and* dependencies (PATCH increase).  However, whenever the
> > > version number signalizes incompatibility (MAJOR) or new features
> > > (MINOR), further inspection of the entire dependency tree is required
> > > because consistency cannot be derived from SemVer versions alone.  The
> > > only definite conclusion that can be made from observing that an
> > > artifact changes its MAJOR or MINOR version, i.e. becomes incompatible,
> > > is that all dependents will need to be updated/ recompiled or that
> > > there's a problem, consistency across artifacts cannot be guaranteed.
> > > I.e. the bubbling scheme, at every individual artifact, sends a signal
> > > when further inspection is required.  This information, however is
> > > binary, and a single counter would suffice to do that.  If patch
> > > counters are desired, one would need two.  The MAJOR and MINOR counters
> > > are redundant.
> > >
> > > Short:
> > > Non-bubbling SemVer tells a developer whether her code will compile with
> > > a given dependency artifact (PATCH and MINOR).  Bubbling SemVer tells a
> > > deployer when a single artifact can be deployed without considering its
> > > environment (PATCH).  Both things exclude each other.  In this setup, I
> > > find the benefit for developers stronger and therefore prefer
> > > non-bubbling SemVer.
> > >
> > > An interesting animal are dependency management poms (BOMs) such as
> > > pom-fiji or pom-mpicbg.  Their `public API' is the composite of  managed
> > > dependencies and therefore it has to bubble the SemVer versions of the
> > > managed dependencies.  This is different from artifacts that consume a
> > > dependency, consuming and managing are different.  I may be wrong, but I
> > > have the impression that these two things often get confused.
> > >
> > > Please let me know your thoughts.
> > >
> > > Thanks,
> > > Stephan
> > >
> > >
> > >
> >
> >


_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel
Reply | Threaded
Open this post in threaded view
|

Re: SemVer policies for ImageJ2 and Fiji

Mark Hiner
Hi Stephan,

>`unintended' behavior (what's that in the first place)

For example, if I have class Foo with method add(int, int) that behaves as one might expect, except for the following edge case: Foo.add(0, -1) = 256. That is unintended behavior. If I fix my implementation so that Foo.add(0, -1) = -1 I have changed behavior but not the traditional definition of public API[1,2]. Thus this change is within the scope of the PATCH version of SemVer.

>A strict definition of the public API would classify a bug-fix that changes the output of a method
>from wrong to correct as a break, i.e. MAJOR upgrade.

> and thanks to the SemVer policy to roll PATCH on MINOR (1.1.5 > 1.2.0)

Just to make sure we're on the same page - when you say "changes the output", I assume you mean "changes the behavior but not the API - i.e. return type is unchanged" because SemVer already requires bug fixes that change API to be MAJOR version bumps.

In that case, I think it would be cleaner to just eliminate the PATCH number - because every bug fix necessarily changes behavior, right? So with this scheme, MAJOR increases = "existing behavior has changed", and MINOR increases = "new behavior is available".

Some problems with creating this versioning scheme:
1. MAJOR versions will increase rapidly. This is aesthetic, but one that people can react very negatively to - and can certainly be confusing if people don't expect MAJOR version bumps to cover bug fixes.
2. It's one more thing for external developers to learn. We can't just say "we use SemVer".
3. Since this is an internal versioning scheme, it may not be easy to compare our versions with external project versions that use SemVer.
4. There will be false negatives for MAJOR version compatibility comparisons (instead of what could be considered false positives with SemVer)

Unfortunately, #1 above alone makes it very unlikely that we would want to adopt this use of version numbers. But what you're trying to do here - ensure compatibility - is fantastic and something that would be great to have.

So let's take a step back and look at what guarantees we do and do not have right now:

+ We have reproducible builds (release couplings, which requires -some- versioning scheme to be used)
+ We have API compatibility (SemVer)

- We do not have strict behavior compatibility
- We do not have dependency compatibility

Behavior and dependency compatibility are very closely related - if they were covered by a versioning scheme, we could automatically answer the question "is it safe to drop in version X to replace version Y?".

However, I do not think we should conflate these concerns with SemVer. Instead, two potential options would be:

1) Continue to use SemVer, accept its limitations, be content with reproducibility.
2) Create a separate versioning scheme that covers behavior and dependency compatibility. Use it in tandem with SemVer.

If anyone can think of other examples of versioned guarantees that would be useful to have, or counter-examples to any claims made here - please share!

Also, please let me know if any of this is confusing and/or additional examples would be useful.

Thanks again for the continued discussion,
Mark

_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel
Reply | Threaded
Open this post in threaded view
|

Re: SemVer policies for ImageJ2 and Fiji

Stephan Saalfeld-2
Hi Mark,

>[...] but not the traditional definition of public API[1,2].

This is what worries me because for most things that are less trivial
than addition, `traditional' definitions are not available, i.e.
`unintended' is not well defined.

> Just to make sure we're on the same page - when you say "changes the
> output", I assume you mean "changes the behavior but not the API - i.e.
> return type is unchanged"

In a typed language yes, what about Python?  What I mean is indeed
change in behavior---correct.  I proposed to treat change in behavior as
change of API because that would enable to reason about compatibility.
If we enable PATCH upgrades to change behavior but only require `compile
time' compatibility (which limits the scope of PATCH to statically typed
and compiled languages which, I think, was not intended), then all we
can reason from seeing a PATCH upgrade is that the stuff will compile,
and somehow run, no guarantees about behavior.  That's not very helpful
because compiling and running stuff isn't expensive to just try.
Behavior, however, is critical and complicated to test, having a
contract that makes guarantees here would indeed be useful.

> In that case, I think it would be cleaner to just eliminate the PATCH
> number - because every bug fix necessarily changes behavior, right? So with
> this scheme, MAJOR increases = "existing behavior has changed", and MINOR
> increases = "new behavior is available".
>

Not necessarily.  Performance improvements, crash-fixes, compatibility
adjustments to upgraded dependencies (!) would fall into PATCH.  But I
see that even crashes could be used by dependents as part of the API...
we're doomed.

> Some problems with creating this versioning scheme:
> 1. MAJOR versions will increase rapidly. This is aesthetic, but one that
> people can react very negatively to - and can certainly be confusing if
> people don't expect MAJOR version bumps to cover bug fixes.

Right---but SemVer is already counterintuitive compared to romantic
versioning where a MAYOR upgrade is associated with a lot of new
features, not breakage.

> 2. It's one more thing for external developers to learn. We can't just say
> "we use SemVer".

Very true.

> 3. Since this is an internal versioning scheme, it may not be easy to
> compare our versions with external project versions that use SemVer.

Why would we want to do this?

> 4. There will be false negatives for MAJOR version compatibility
> comparisons (instead of what could be considered false positives with
> SemVer)

I wouldn't call them false but conservative or overly cautious.  It
would tell you when compatibility is not guaranteed which is useful.

>
> Unfortunately, #1 above alone makes it very unlikely that we would want to
> adopt this use of version numbers.

I understand and fully agree.  But SemVer with PATCHES of `unintended
behavior' or other sentimental or romantic versioning schemes (I am
stealing these terms from

https://gist.github.com/jashkenas/cbd2b088e20279ae2c8e

and

http://sentimentalversioning.org/

which are both exciting reads) aren't useful to reason about
compatibility.  We should therefore not use them to reason about
compatibility which you correctly state in the following...

> But what you're trying to do here -
> ensure compatibility - is fantastic and something that would be great to
> have.
>
> So let's take a step back and look at what guarantees we do and do not have
> right now:
>
> + We have reproducible builds (release couplings, which requires -some-
> versioning scheme to be used)
> + We have API compatibility (SemVer)
>

No---there is no strict API compatibility when using PATCH to fix
`unintended behavior'.  Happened with ImageJ many times over the last
years: API compile-time compatible that would qualify as PATCH upgrade,
behavior different, plugins delivering crazy output, discovered only
much later through bug reports.

> - We do not have strict behavior compatibility
> - We do not have dependency compatibility
>
> Behavior and dependency compatibility are very closely related - if they
> were covered by a versioning scheme, we could automatically answer the
> question "is it safe to drop in version X to replace version Y?".
>

Yes.

> However, I do not think we should conflate these concerns with SemVer.

I could not agree more.

> Instead, two potential options would be:
>
> 1) Continue to use SemVer, accept its limitations, be content with
> reproducibility.

Or any other versioning scheme that people like.  My current impression
is that SemVer doesn't help much.  I will keep using it, but it isn't
any better than any other scheme that assigns keys to versions and
enables to order them.

> 2) Create a separate versioning scheme that covers behavior and dependency
> compatibility. Use it in tandem with SemVer.
>

You're absolutely right, but---phew---that's probably asking too much of
people that are already unwilling to adopt to something as simple as
SemVer.

My current thinking is that we should require contributors to use a
versioning scheme that can be sorted correctly by

sort -V

Anything else, i.e. compatibility guarantees, aren't possible to express
with a simple number and require runtime testing (e.g. unit-tests
promise this int an ideal world).  We should therefore never drop any
upgrade without significant attempt to test.  We could assume that a
maven build includes sufficient [unit-]testing to enable deployment if
it succeeds (this is not true in reality but it is a meaningful
contract).  I.e. artifacts can be deployed if they were explicitly built
against upgraded dependencies.  In our heterogeneous environment,
however, it is unrealistic to expect all contributors to adapt their
code in time.  Instead, we should try to build pom-fiji including all
downstream modules *overriding* or *upgrading* their dependency versions
to the highest possible.  Do you have something like this already or do
you have an idea how that could be done?

Thanks for all the energy that you're putting into this!

Cheers,
Stephan



> If anyone can think of other examples of versioned guarantees that would be
> useful to have, or counter-examples to any claims made here - please share!
>
> Also, please let me know if any of this is confusing and/or additional
> examples would be useful.
>
> Thanks again for the continued discussion,
> Mark
>
> [1]
> http://en.wikipedia.org/wiki/Application_programming_interface#API_in_object-oriented_languages
> [2]
> http://stackoverflow.com/questions/2954372/difference-between-spi-and-api


_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel
Reply | Threaded
Open this post in threaded view
|

Re: SemVer policies for ImageJ2 and Fiji

Mark Hiner
Hi Stephan,

>Behavior, however, is critical and complicated to test, having a
>contract that makes guarantees here would indeed be useful

A contract that guarantees this would be amazing. I think we are on the same page, but just wanted to illustrate how bugs make this an intractable problem for versioning:

If bug fixes that change behavior would cause MAJOR version bumps, the introduction of bugs that change behavior must also cause a MAJOR version bump. Thus you need to know if you have added a bug or not before doing a release. Failure to identify a bug means the release of versions that do not follow the versioning scheme, which means we can not actually use the version to reason about compatibility.

>> 3. Since this is an internal versioning scheme, it may not be easy to
>> compare our versions with external project versions that use SemVer.
>
>Why would we want to do this?

My original thought was dependency convergence - but for that the versions comparisons are between artifacts with the same versioning system, so all you need is internal consistency. So this may not actually be important.

>http://sentimentalversioning.org/

This page is great. I either hadn't read it or had forgotten it, so thanks for sharing!

> https://gist.github.com/jashkenas/cbd2b088e20279ae2c8e

I think the author of this post slightly misunderstands the use of SemVer. I completely agree with some of the content ('If you expect SemVer to solve your problems for you, you will be disappointed'). But the author seems to blame SemVer for failing to account for changes in behavior, when really the failure was in their use of SemVer.

In the intro of http://semver.org/ is the phrase: "For this system to work, you first need to declare a public API." If a developer doesn't declare API to include behavior then they can't expect behavior to respect SemVer. If they decide API does include behavior and then follow SemVer, then behavior would be guaranteed between compatible versions - assuming they updated the version appropriately when changing behavior.

>Instead, we should try to build pom-fiji including all
>downstream modules *overriding* or *upgrading* their dependency versions
>to the highest possible.  Do you have something like this already or do
>you have an idea how that could be done?

We have been throwing around the phrase "Melting Pot" for a while. This is our dream: to have a Jenkins job that pulls the full SciJava software stack and tests everything together based - i.e. build Fiji's pom and test. This would check for dependency convergence as you outlined, and make whatever guarantees of behavior we can automate.

There is a script started to do this, but no tests incorporating it that I know of yet. But it sounds like we're on the same page, that this is how our software should be tested.

With this in mind, my proposal is:

* Define "API" for our software to be limited to public interfaces, classes and methods, but not behavior. This is what will be covered by SemVer.
* Develop Melting Pot tests to validate behavior and dependency convergence

Unless there are any objections to this, I will update http://imagej.net/Architecture#Versioning to explicitly define API as such, and summarize the concerns that have been raised in this discussion - including how the Melting Pot will make everything happy.

>we're doomed.

Probably!

>Thanks for all the energy that you're putting into this!

You too!

Best,
Mark

On Fri, Mar 27, 2015 at 10:06 PM, Stephan Saalfeld <[hidden email]> wrote:
Hi Mark,

>[...] but not the traditional definition of public API[1,2].

This is what worries me because for most things that are less trivial
than addition, `traditional' definitions are not available, i.e.
`unintended' is not well defined.

> Just to make sure we're on the same page - when you say "changes the
> output", I assume you mean "changes the behavior but not the API - i.e.
> return type is unchanged"

In a typed language yes, what about Python?  What I mean is indeed
change in behavior---correct.  I proposed to treat change in behavior as
change of API because that would enable to reason about compatibility.
If we enable PATCH upgrades to change behavior but only require `compile
time' compatibility (which limits the scope of PATCH to statically typed
and compiled languages which, I think, was not intended), then all we
can reason from seeing a PATCH upgrade is that the stuff will compile,
and somehow run, no guarantees about behavior.  That's not very helpful
because compiling and running stuff isn't expensive to just try.
Behavior, however, is critical and complicated to test, having a
contract that makes guarantees here would indeed be useful.

> In that case, I think it would be cleaner to just eliminate the PATCH
> number - because every bug fix necessarily changes behavior, right? So with
> this scheme, MAJOR increases = "existing behavior has changed", and MINOR
> increases = "new behavior is available".
>

Not necessarily.  Performance improvements, crash-fixes, compatibility
adjustments to upgraded dependencies (!) would fall into PATCH.  But I
see that even crashes could be used by dependents as part of the API...
we're doomed.

> Some problems with creating this versioning scheme:
> 1. MAJOR versions will increase rapidly. This is aesthetic, but one that
> people can react very negatively to - and can certainly be confusing if
> people don't expect MAJOR version bumps to cover bug fixes.

Right---but SemVer is already counterintuitive compared to romantic
versioning where a MAYOR upgrade is associated with a lot of new
features, not breakage.

> 2. It's one more thing for external developers to learn. We can't just say
> "we use SemVer".

Very true.

> 3. Since this is an internal versioning scheme, it may not be easy to
> compare our versions with external project versions that use SemVer.

Why would we want to do this?

> 4. There will be false negatives for MAJOR version compatibility
> comparisons (instead of what could be considered false positives with
> SemVer)

I wouldn't call them false but conservative or overly cautious.  It
would tell you when compatibility is not guaranteed which is useful.

>
> Unfortunately, #1 above alone makes it very unlikely that we would want to
> adopt this use of version numbers.

I understand and fully agree.  But SemVer with PATCHES of `unintended
behavior' or other sentimental or romantic versioning schemes (I am
stealing these terms from

https://gist.github.com/jashkenas/cbd2b088e20279ae2c8e

and

http://sentimentalversioning.org/

which are both exciting reads) aren't useful to reason about
compatibility.  We should therefore not use them to reason about
compatibility which you correctly state in the following...

> But what you're trying to do here -
> ensure compatibility - is fantastic and something that would be great to
> have.
>
> So let's take a step back and look at what guarantees we do and do not have
> right now:
>
> + We have reproducible builds (release couplings, which requires -some-
> versioning scheme to be used)
> + We have API compatibility (SemVer)
>

No---there is no strict API compatibility when using PATCH to fix
`unintended behavior'.  Happened with ImageJ many times over the last
years: API compile-time compatible that would qualify as PATCH upgrade,
behavior different, plugins delivering crazy output, discovered only
much later through bug reports.

> - We do not have strict behavior compatibility
> - We do not have dependency compatibility
>
> Behavior and dependency compatibility are very closely related - if they
> were covered by a versioning scheme, we could automatically answer the
> question "is it safe to drop in version X to replace version Y?".
>

Yes.

> However, I do not think we should conflate these concerns with SemVer.

I could not agree more.

> Instead, two potential options would be:
>
> 1) Continue to use SemVer, accept its limitations, be content with
> reproducibility.

Or any other versioning scheme that people like.  My current impression
is that SemVer doesn't help much.  I will keep using it, but it isn't
any better than any other scheme that assigns keys to versions and
enables to order them.

> 2) Create a separate versioning scheme that covers behavior and dependency
> compatibility. Use it in tandem with SemVer.
>

You're absolutely right, but---phew---that's probably asking too much of
people that are already unwilling to adopt to something as simple as
SemVer.

My current thinking is that we should require contributors to use a
versioning scheme that can be sorted correctly by

sort -V

Anything else, i.e. compatibility guarantees, aren't possible to express
with a simple number and require runtime testing (e.g. unit-tests
promise this int an ideal world).  We should therefore never drop any
upgrade without significant attempt to test.  We could assume that a
maven build includes sufficient [unit-]testing to enable deployment if
it succeeds (this is not true in reality but it is a meaningful
contract).  I.e. artifacts can be deployed if they were explicitly built
against upgraded dependencies.  In our heterogeneous environment,
however, it is unrealistic to expect all contributors to adapt their
code in time.  Instead, we should try to build pom-fiji including all
downstream modules *overriding* or *upgrading* their dependency versions
to the highest possible.  Do you have something like this already or do
you have an idea how that could be done?

Thanks for all the energy that you're putting into this!

Cheers,
Stephan



> If anyone can think of other examples of versioned guarantees that would be
> useful to have, or counter-examples to any claims made here - please share!
>
> Also, please let me know if any of this is confusing and/or additional
> examples would be useful.
>
> Thanks again for the continued discussion,
> Mark
>
> [1]
> http://en.wikipedia.org/wiki/Application_programming_interface#API_in_object-oriented_languages
> [2]
> http://stackoverflow.com/questions/2954372/difference-between-spi-and-api



_______________________________________________
ImageJ-devel mailing list
[hidden email]
http://imagej.net/mailman/listinfo/imagej-devel